#pyqt | Logs for 2019-02-03

Back
[02:29:05] -!- Dave_Elec has joined #pyqt
[03:40:03] -!- Mrokii has quit [Ping timeout: 245 seconds]
[03:40:48] -!- Mrokii has joined #pyqt
[03:44:56] -!- Dave_Elec has quit [Ping timeout: 240 seconds]
[04:54:53] -!- BPL has joined #pyqt
[07:45:53] -!- mintograde has joined #pyqt
[07:58:33] -!- Dave_Elec has joined #pyqt
[08:04:36] -!- TechSmurf has quit [Ping timeout: 252 seconds]
[08:11:47] -!- Dave_Elec has quit [Ping timeout: 240 seconds]
[09:36:31] -!- Dave_Elec has joined #pyqt
[09:53:02] -!- mintograde has quit [Read error: Connection reset by peer]
[09:53:23] -!- Dave_Elec has quit [Ping timeout: 245 seconds]
[09:55:32] -!- Dave_Elec has joined #pyqt
[10:10:56] -!- Dave_Elec has quit [Ping timeout: 250 seconds]
[10:21:27] -!- mintograde has joined #pyqt
[11:56:30] -!- Dave_Elec has joined #pyqt
[12:43:28] -!- Dave_Elec has quit [Ping timeout: 250 seconds]
[12:46:43] -!- Dave_Elec has joined #pyqt
[12:51:21] -!- mintograde has quit [Quit: Leaving]
[13:02:52] -!- mintograde has joined #pyqt
[14:32:40] -!- Dave_Elec has quit [Ping timeout: 250 seconds]
[14:32:56] -!- Dave_Elec has joined #pyqt
[15:13:23] -!- Dave_Elec has quit [Ping timeout: 245 seconds]
[15:33:20] -!- Dave_Elec has joined #pyqt
[15:56:26] -!- Dave_Elec has quit [Ping timeout: 240 seconds]
[16:07:58] -!- Dave_Elec has joined #pyqt
[17:05:38] -!- qt-perf has joined #pyqt
[17:08:11] <qt-perf> I have a PyQt application that collects data over a socket and does some transformations on it to display to a user. I'm trying to find the performance bottlenecks in it and have been using cProfile. I can see that all of my code functions calls eat up ~60% of application runtime, and of course _exec() is listed as taking up nearly all of application runtime, since it's the event loop.
[17:08:30] <qt-perf> I'm trying to push my system to the limit, but still don't see my functions getting any more time.
[17:08:34] <qt-perf> What all does _exec() do?
[17:08:41] <qt-perf> Could I have a bottleneck in QML??
[17:10:21] <qt-perf> Currently because my socket controller and data processing code runs on a QThread, I'm figuring I should move the socket controller onto it's own process and then spawn processes to handle the data and perform transformations, and then feed all of that back to PyQt, to get a performance gain.
[17:10:35] <qt-perf> But before I do that, I'm looking for ways to improve performance with the current structure.
[17:30:28] <altendky> qt-perf: I wouldn't expect a thread to help anything. You only get one CPU worth of time for python code per process. Are you using the python socket module? Or qt? The qt stuff would already be async
[17:30:45] <altendky> qt-perf: also, what processing are you doing?
[17:33:29] <qt-perf> altendky: Yeah, I'm more interested in knowing what _exec() is doing when it's not running my code; is it just looping with nothing to do?
[17:33:41] <Avaris> qt-perf: something like that
[17:33:55] <qt-perf> altendky: I'm doing peak detection on some ultrasound signals and measuring thicknesses of metals
[17:34:23] <altendky> qt-perf: just python code? Numpy?
[17:34:38] <Avaris> i'd suggest profiling your non-qt parts independently first
[17:34:46] <altendky> And no, if the loop just spun you'd always be at 100% cpu
[17:34:48] <qt-perf> altendky: Python socket module, and the code for processing is a lot of numpy and numba
[17:35:18] <altendky> qt-perf: so numpy will release the Gil often so having that in a thread could get you more CPU time
[17:35:29] -!- TechSmurf has joined #pyqt
[17:35:31] <altendky> qt-perf: but I'd definitely be using twisted for networking
[17:35:49] <altendky> You can deferToThread for the numpy stuff
[17:35:52] <qt-perf> altendky: Interesting. I looked at cProfile and I didn't really see any time spent in my socket code.
[17:35:56] <Avaris> (depending on the thing numpy does, not everything releases GIL)
[17:36:32] <altendky> qt-perf: sure, it's just error prone and you don't have any async tooling then
[17:36:40] <altendky> "often"
[17:37:01] <qt-perf> altendky: Cool, I'll take a look at twisted
[17:37:33] <qt-perf> I'm thinking I should have one process that connects to my data acquisition unit, and then farm the data out to a bunch of processes to handle thoughput
[17:37:44] <altendky> qt-perf: you can integrate with pyqt via qt5reactor. And no, I don't suggest twisted for CPU usage reduction. Just higher level networking tooling
[17:38:31] <Avaris> qt-perf: out of curiosity, what kind of performance issue are we talking about?
[17:39:57] <qt-perf> Avaris: Based on my profiling, my signal processing code can achieve a throughput of ~5 kHz signals processed / sec. However in my current PyQt application, I'm seeing a much lower throughput.
[17:40:17] <altendky> qt-perf: how long is a 'task'?
[17:40:30] <altendky> That's the 5khz?
[17:40:43] <qt-perf> altendky: Yes
[17:40:45] <altendky> 200us per?
[17:41:01] <Avaris> lower as in...?
[17:41:15] <qt-perf> altendky: I measure longer times when looking at cProfile than with my benchmarks
[17:41:31] <qt-perf> Roughly 2x longer
[17:41:36] <altendky> qt-perf: sure, but each unit of work or whatever
[17:41:50] <Avaris> some overhead isn't unexpected
[17:42:18] <Avaris> also can the socket deliver that much data?
[17:42:54] <qt-perf> Avaris: I'm not sure, I haven't gotten nearly that throughput yet
[17:43:13] <qt-perf> I'm planning on pulling this code out of the PyQt application and running it on its own to test some of this
[17:43:39] <Avaris> that's probably best
[17:44:00] <qt-perf> I was judging by how much data I can handle over Ethernet; should a socket be able to handle 60 MB / s? That's what I expect at the throughput I'm aiming for.
[17:44:19] <qt-perf> Target throughput is 9.6 kHz, and my application is far from it right now
[17:45:10] <altendky> qt-perf: that's half of a gigabit connection...
[17:45:30] <altendky> Seems pretty optimistic
[17:46:28] <qt-perf> altendky: I don't know that much about networking, but I figured that if the hardware supports Gigabit Ethernet, I should be able to find a way to get this data processed in my code
[17:46:53] <qt-perf> altendky: Where would I be likely to have a bottleneck? Is this too much for a single socket to handle?
[17:48:40] <altendky> qt-perf: that sounds optimistic for a single connection (the cable) without a fairly specific setup. But i can't say I've tested the limits. There's just several layers of overhead.
[17:49:40] <qt-perf> altendky: Okay. Yeah I haven't tested this much throughput yet either. However the Gigabit connection will be almost exclusively reserved for this, and I will only be using half of it.
[17:50:07] <qt-perf> altendky: Wasn't sure if you saw an issue with the software side, e.g. an implementation with socket or with twisted.
[17:58:22] <altendky> qt-perf: people just usually mess up socket code and end up rewriting existing features that networking libs do. And threading in Python often isn't really great anyways, so async is a good choice
[17:58:59] <altendky> qt-perf: but the gigabit is the electrical signals I thought, not actual useful throughput of an application.
[18:03:41] <qt-perf> altendky: Right. I've just been trying to get my arms around the throughput here. If my signal processing code can do 5 kHz, and I run it on two cores, processing is covered. I also have to receive all of that data and farm it to the workers, and then collect the data and write it to disk. I'm going to be testing on a 4-core machine with max turbo frequency of 4.8 GHz, and it won't be doing much other than handling this data.
[18:09:31] <altendky> qt-perf: what does your GUI do?
[18:10:18] <qt-perf> qt-perf: Plots a subset of the ultrasound signals and reports thickness readings
[18:10:37] <qt-perf> *altendky
[18:11:34] <qt-perf> Honestly the whole application is running on a single core right now, so by adding 3 more cores and moving most of the "work" to these other cores and leaving the PyQt application with less work, I think it will be fine.
[18:12:16] <qt-perf> The PyQt part should be focused on giving a subset of the data to the user; I'm not even sure that I should use the Qt event system to emit at a rate of 9.6 kHz.
[18:12:33] <qt-perf> I was planning on moving all of that away from the heart of the Qt application
[18:18:26] <BPL> any idea how to create an "alt+enter" shortcut? this one isn't working https://bpaste.net :/
[18:19:56] <BPL> help on shortcut is this `QShortcut(Union[QKeySequence, QKeySequence.StandardKey, str, int], QWidget, member: PYQT_SLOT = 0, ambiguousMember: PYQT_SLOT = 0, context: Qt.ShortcutContext = Qt.WindowShortcut)`
[18:21:14] <BPL> that means I can use a much simpler syntax like this https://bpaste.net ... but still, "alt+enter" seems to be not a valid string
[18:22:07] <Avaris> <trivia>enter and return are different keys</trivia>
[18:22:11] <altendky> BPL: so don't use the strings
[18:23:25] <BPL> Avaris: Yeah, that's correct, keyPressEvent has reported me these 2 ints, 16777220
[18:23:25] <BPL> 16777251 , now i need to know how to mix both
[18:23:45] <Avaris> enter is the one next to the numpad
[18:24:40] <BPL> Avaris: ok, you're right "alt+enter" is by using the numpad one, do you know how's called the big one? :'D
[18:25:12] <BPL> enter_big=16777220 enter_numpad=16777221
[18:25:43] <Avaris> the one in the "normal" part is return. as in carriage-return, a.k.a \r
[18:26:09] <BPL> ty! it works now :)
[18:26:48] <BPL> damm these spanish keyboard, both are called "Entrar"=Enter , so confusing :O
[18:28:30] <Avaris> i didn't see "return" as a name for a long time on keyboards. usually a left arrow thing or sometimes enter + arrow
[18:35:05] <BPL> this logitech keyboard is really funny as it mixes english+spanish keys confusing to clueless users like me, that said, next time I'll check directly http://doc.qt.io
[18:36:03] <BPL> not sure if it's possible to get the string out of Qt.Key though... that'd be interesting
[18:37:05] <BPL> found it --> >>> QKeySequence(Qt.Key_Tab).toString() <--
[18:37:09] <BPL> moving on
[18:38:05] <BPL> >>> QKeySequence(16777220).toString() == "Return" ; >>> QKeySequence(16777221).toString() == "Enter" . Just to be extra sure ;)
[19:32:51] -!- Dave_Elec has quit [Ping timeout: 268 seconds]
[20:05:14] -!- BPL has quit [Quit: Leaving]
[22:18:19] -!- kdas_ has joined #pyqt
[22:19:45] -!- kushal has quit [Ping timeout: 256 seconds]
[22:21:12] kdas_ is now known as kushal
[22:27:15] -!- JanC has quit [Remote host closed the connection]
[22:27:52] -!- JanC has joined #pyqt
[22:51:50] -!- mintograde has quit [Ping timeout: 246 seconds]
[22:52:15] -!- Mrokii has quit [Ping timeout: 244 seconds]
[23:06:21] -!- Mrokii has joined #pyqt
[23:45:08] -!- Dave_Elec has joined #pyqt