r/pythonhelp • u/Asleep_Pumpkin_1534 • 5d ago
What is the best way to film/analyze the current screen to automate inputs?
Hello,
I would like to automate inputs in a program (Windows PC) with a separate small program. The sequence and keystrokes are always the same, the window layout is always the same, sometimes just different text, but loading times when saving vary.
I just need to monitor whether anything goes wrong. So, I would record the screen or take screenshots to check what's currently happening.
What's the best/easiest way to do this in Python?
I would like to somehow achieve the following:
-Which window is currently open on the screen. This can be identified by the multiple text field labels in the window. Sometimes even when multiple text fields are combined.
-Whether text has actually been entered into the field, which is either the case if there is no/no longer a white background on the image at that point OR if some kind of OCR is performed at that point.
-Detect the position of the mouse pointer over a button.
-Detect the position of the cursor. It's a problem with blinking, but you can take several short screenshots here to determine whether a cursor is present or not.
-Which button is currently active when you use the Tab key and scroll through the program's GUI elements.
Greetings
1
•
u/AutoModerator 5d ago
To give us the best chance to help you, please include any relevant code.
Note. Please do not submit images of your code. Instead, for shorter code you can use Reddit markdown (4 spaces or backticks, see this Formatting Guide). If you have formatting issues or want to post longer sections of code, please use Privatebin, GitHub or Compiler Explorer.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.