Need help with touch screen macro

Issue:
Hello, I’m trying to do a script to click on a part of the touch screen to avoid switching between gamepad and mouse in Zelda OOT.
My script use image recongnition to find the right button and I will asign a pad button do do the script.
The problem is that Citra does not seems to support pyautogui. Do you have I idea how I can do that? Here is ma Python code for testing.

import pyautogui
import time

time.sleep(3)

Recherche puis clic bouton Item

itemLoc = pyautogui.center(pyautogui.locateOnScreen (‘item.png’, confidence=0.5))
pyautogui.click(itemLoc.x, itemLoc.y)

System Information

  • Operating System: Windows 10
  • CPU: AMD FX 8350
  • GPU: Geforce 660ti
  • Citra Version (found in title bar): Nightly 1203
  • Game: Zelda OOT 3D

Diagnostic Log
Upload your log file as an attachment by dragging & dropping.

In order to save a copy of the log, follow this guide:
https://community.citra-emu.org/t/how-to-upload-the-log-file/296

In order for us to provide better support, we need to see the log generated by Citra. This guide will walk you through how you can obtain the log file: How to Upload the Log File.

just use joytokey and map the button with a mouse click + coordinates of the mouse, im using it so that when i click L2 or ZL it clicks the ocarina and R2 or ZR to click items, select to view mode, and R analog to click I and II buttons

Hoo I don’t knew Joy2Key could do that, wil give it a try. Can it do image recognition too? Because I’m playing on different layout (portrait via moonlight when not a home and landscape when at home)

so youre playing on mobile

I’m playing on PC through my mobile with a ps3 pad :slight_smile:

Just tried, it does not works. The click is made on the return to desktop button. I think the fact that I use Nvidia Gamestream mess something with resolution and mouse coordinates

that’s not a very practical idea, there are better ways to do that, either by finding the addresses of those buttons and call them or by sending a click event to the buttons position on screen.

I have no idea how to do that :frowning: I wanted to made an image recognition to be able to change screen configuration, but even without that, just click at a fixed position on screen, it does not work.

Hello. I’ve done a few tests and it’s not working with joy2key, it does not click the right place. Maybe because double screen or maybe because of streaming.
Anyway it’s not a solution for me because sometimes I stream 1080p and sometimes 360 or 480 (network issues in train and tramway), so the mouse coordinates changes.
It would be a good feature for citra to assign tactile screen zone to key :slight_smile:

It may be easier by using AHK, for instance this simply code (source) works just fine on citra:

x := ;place your desired coordinates in these two variables.
y := 

f1::
 {
   mousegetpos, start_x, start_y
   mouseclick, left, %x%, %y%, 1, 0
   mousemove, %start_x%, %start_y%, 0
 }
return

and you can get the windows x,y,w,h with this function:
https://autohotkey.com/docs/commands/WinGetPos.htm#Example

now you can know the windows coordinates before sending the click event and properly calculate the desirable click position (i guess).

Hello,

That’s what I have done just have two scripts one for 720p and one for 1080p.
When streaming at 360 or 480, the real resolution is 720p