MyoChallenge 25'/MyoSuite Q&A

[cherylwang20]

Hey challenger!

Welcome to the official discussion thread for MyoChallenge '25. This year's challenge pushes the boundaries of agility and dexterity, with new tasks in soccer ⚽ and table tennis 🏓 featured in both the Manipulation and Locomotion tracks to push the limits of robust control and adaptive learning.

If you have any questions or need clarification about the challenge or the use of MyoSuite, this is the place to find answers. You’re also encouraged to use this space to team up with others to tackle the challenge together if you’re looking for collaborations!

Feel free to ask and interact with the community, be respectful of other participants, and good luck with the challenge! 🦾

[Mehshida]

Hi myoteam,
I was looking for the baselines for the myosoccer, and couldn't locate them. Can you help?

[cherylwang20]

Hi Mehshida,

thank you for your enthusiasm. Currently the baseline for both tracks are not released yet and will be likely open to public in the next few weeks or so. During the mean time feel free to try it out on your own. Also looping in the lead for soccer @elladyr for more information.

-Cheryl

[sharinka0715]

Dear Myochallenge Organizers,

The submission window for the 2025 competition is not yet open, but I see that the latest version of MyoSuite has already integrated the relevant environments and documentation. I would like to inquire if the current version represents the final settings for Stage 1 of the competition, or if there will be further adjustments.

If adjustments are still possible, I have a few minor concerns I'd like to share:

1. The two current competition environments have no randomness during initialization. Since MuJoCo is a deterministic physics engine, a participant's policy could achieve a 100% success rate by simply replaying the same sequence of actions. This would render the "success rate" metric meaningless, effectively turning the competition into a challenge to minimize muscle activation. Have you considered introducing some form of randomness into the environments, or adding other trajectory quality metrics (e.g., the ball's flight time, landing position, etc.)?

2. In the current pingpong-v0 environment, the observation space appears to be missing the racket's orientation (quaternion) and angular velocity. Would it be possible to add this information to the observations? Could the ball's quaternion and angular velocity also be included (although I'm not certain if a spinning ball would affect the trajectory)?

Thank you for your time and consideration. I look forward to your response.

[cherylwang20]

Dear Jun,

Thanks for your question. We would like to clarify that changes will indeed be introduced when EvalAI opens (i.e., stage 1).

The two current competition environments have no randomness during initialization. Since MuJoCo is a deterministic physics engine, a participant's policy could achieve a 100% success rate by simply replaying the same sequence of actions. This would render the "success rate" metric meaningless, effectively turning the competition into a challenge to minimize muscle activation. Have you considered introducing some form of randomness into the environments, or adding other trajectory quality metrics (e.g., the ball's flight time, landing position, etc.)?

We are aware of your concern, and we plan to add some slight variations in the environment during phase 1. Nevertheless please note that phase 1 generally allows people of various level to enter and participate and we will be introduce significantly more difficult varations in phase 2.

In the current pingpong-v0 environment, the observation space appears to be missing the racket's orientation (quaternion) and angular velocity. Would it be possible to add this information to the observations? Could the ball's quaternion and angular velocity also be included (although I'm not certain if a spinning ball would affect the trajectory)?

We plan to add the racket's orientation to the observation as seen during baseline policy training. Can you elaborate why is the ball's quaternion and angular velocity necessary?

[sharinka0715]

Thank you for your reply! I'm looking forward to the opening of the EvalAI evaluation window and the release of the latest environment version.

Regarding the issue of orientation and angular velocity: the orientation of the ping pong ball should be irrelevant, as it is treated as a standard sphere in the simulator, making any orientation equivalent to the default one.

As for the angular velocity, while I'm not sure if the MuJoCo simulator models this effect, based on real-world table tennis experience, a ball's trajectory changes distinctively when it hits the table with a certain spin. Professional players use special techniques to hit the ball with angular velocity (spin), making the trajectory difficult for their opponents to predict and return.

However, considering the specific nature of the MyoChallenge—a single strike where the ball initially has no angular velocity—this property probably doesn't need to be added to the observation space. We should focus more of our efforts on the control of the musculoskeletal system.

[siyuan-liu-casia]

Hi MyoChallenge Team,
I would like to inquire whether modifications to the XML are permitted. Specifically, is it allowed to add new sites for observation, or must the observation settings remain identical to those provided in the original environment?

[cherylwang20]

Hi Siyuan,

You are free to make any modifications during your own training; this includes changing the XMLs, rewards, etc.

However, the observation will remain the same for everyone. Which observation are you thinking of adding?

-Cheryl

[siyuan-liu-casia]

Hi Cheryl,

This means our observations should all be ['pelvis_pos', 'body_qpos', 'body_qvel', 'ball_pos', 'ball_vel', 'paddle_pos', "paddle_vel", 'reach_err', "touching_info"], correct?

As you mentioned above, there's a plan to add the racket's orientation to the observations. Does this mean our observation setup will change? Or can observations be set to any data that can be directly read from the original XML file? I'm not sure if I'm expressing this clearly. Thank you very much for your feedback.

By the way, I'm also thinking about potentially including the the ball-paddle relative velocity as observations, though they may not actually be useful.

-siyuan liu

[cherylwang20]

Yes, please do expect the observation space for table tennis to change slightly during the official phase 1 release.

You are free to use add/use any sites during the reward training, but your final observation space could only be a subset of the given observations. If you deem additional observation is absolutely necessary, please let us know.

Hope this helps!

[timmmml]

Dear MyoSuite Team,

I'm looking to potentially participate MyoChallenge 2025 with several teammates. Is there any constraint on teaming?

Thanks!

[cherylwang20]

We should be following the rules from past years, that is:

Team formation / Eligibility: There is no limit to the team size and the affiliation: participants from both academic and industrial institutions are welcome. Each participant will be allowed to join only one team during registration and this cannot be changed during the competition.

[siyuan-liu-casia]

Hi, cheryl, sorry to bother you again, can participants register for different tracks, such as joining one team for table tennis and another team for soccer? Or participant is allowed to compete in different tracks only within the same team.

[cherylwang20]

Hi siyuan,

yes you are allowed to join two different teams for each track. Although you might need to use a different email if the system doesn’t allow the same account to register in two different teams.

-Cheryl

[K-y-jin]

Dear MyoSuite Team,

I noticed that the number of joints in the "pingpong_v0" environment is 58, which differs from the 86 joints mentioned in the documentation.
Could you please help me understand what I might be missing?

Here is my code.
.............................
import gymnasium as gym
import myosuite

env = gym.make("myoChallengePingpong-v0")

print("-----Observation Space---------")
print(env.observation_space)

print("-----Action Space---------")
print(env.action_space)

obs, info = env.reset()
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)

print("-----Info---------")
print(f"Joint Positions: {info['obs_dict']['body_qpos'].shape}")
print(f"Paddle Possitions: {info['obs_dict']['paddle_pos'].shape}")
print(f"Paddle Reaching Error: {info['obs_dict']['reach_err'].shape}")
print(f"Muscle Activations: {info['obs_dict']['act'].shape}")
print(f"Touching Information: {info['obs_dict']['touching_info'].shape}")

[cherylwang20]

Thanks for pointing this out! You are absolutely right on this. The additional 28 joints are based the qpos of three objects and root of the body. This was added separately to other parts of the observation instead of directly within body_qpos.

We will correct this in the documentation of the upcoming release and thank you for flagging this issue.

[roha913]

Hi!
My friend and I are new to MyoChallenge. I was wondering, are we allowed to make further submissions during Phase 2? Or is Phase 2 only for submissions we make in August?

[Lee-Jun-Hyuk-37]

@cherylwang20

Hi Cheryl,

Thank you for your kind explanation. Just to confirm, would it be possible to skip Phase 1 and only participate in Phase 2 after September?

-Jun Hyuk

[cherylwang20]

Hi Jun Hyuk,

We have now specified that you will need to achieve at least a 10% performance in Phase 1 to participate in Phase 2 on EvalAI homepage. However, the conditions for P1 are much lower than P2, with minimum variations in initial conditions. So ideally, your policy for P2 will generalize well to P1.

-Cheryl

[Lee-Jun-Hyuk-37]

@cherylwang20

Then, does it mean that teams which fail to achieve a 10% performance by the end of August will not be allowed to participate in Phase2 starting in September?

[cherylwang20]

No, Phase 1 will remain open until the end of Phase 2. This means you can still join the challenge in September or October and submit to Phase 1 even after Phase 2 has started. Does this clarify your concern?

[Lee-Jun-Hyuk-37]

Thank you for your kind reply!

[asdspal]

Hi, the discord link on the website is invalid. Can you update and share the new link?

[cherylwang20]

Hi,

Can you try again on this website: https://sites.google.com/view/myosuite/myochallenge/myochallenge-2025. If not you can also try the following link: https://discord.gg/vHseBMUbea

[siyuan-liu-casia]

hi!
When following the submission tutorial, I executed source ./setup_no_docker.sh, which automatically installed myosuite=2.9.0. However, the final release environment should use myosuite=2.10.0. Is that correct?

[cherylwang20]

Hi Siyuan, thanks for the posting. Please use myosuite==2.10.0 for P1!

We are double checking on the source ./setup_no_docker.sh, but the image should be pointing to the latest main on myosuite automatically.

[siyuan-liu-casia]

Hi Cheryl, it seems that the latest main branch on the MyoSuite GitHub repo is still at version 2.9.0, while the pip version of MyoSuite is at 2.10.0. In setup_no_docker.sh, it simply pip installs from the MyoSuite GitHub repository.

[Vittorio-Caggiano]

Hi Cheryl, it seems that the latest main branch on the MyoSuite GitHub repo is still at version 2.9.0, while the pip version of MyoSuite is at 2.10.0. In setup_no_docker.sh, it simply pip installs from the MyoSuite GitHub repository.

@siyuan-liu-casia just checked and there is a bug as the github version didn't get updated. Thanks for pointing this out

[sharinka0715]

Hello,

I was wondering when the participants can start submitting their phase 1 policies on EvalAI. Is it already open and I might have missed the correct submission method?

Currently, I don't see any phase number listed for submission on the EvalAI page. Also, it seems like the myochallenge2025_eval repository might still contain some leftovers from the 2024 challenge template.

I'm not sure if submissions are open for participants at this time. Could you please clarify?

Thank you!

[elladyr]

Hi @sharinka0715, we are currently experiencing some issues with the EvalAI submission system. I would ask for your patience while we work with the EvalAI guys to fix it. Thank you.

By the way, could you also point out where are the leftovers from MyoChallenge 2024? That would help us fix it quickly. Thank you

[sharinka0715]

Of course. You can search "2024" or "mani", "loco" keywords in the repository, and then you would find some leftovers. They mainly exist in some readme files or scripts (setup.sh).

[cherylwang20]

Hi @sharinka0715, Thanks for pointing this out. We have tried our best to remove all previous leftovers from MC24 in the latest commits.

Please stay tuned for the opening of the leaderboard as we work with EvalAI, and we apologize again for this delay.

[hannita1]

Hello everyone,
I tried submitting a solution via GitHub actions but the Workflow keeps failing with the Error message: "Error: Sorry, this is not a docker based challenge.". I also tried running the TableTennis Random Workflow and the same thing happened.
Are submissions not yet open?

Thank you!

[elladyr]

Hi @hannita1 , we are currently experiencing some issues with the EvalAI submission system. Thank you for your patience while we are working it out with the EvalAI guys.

[siyuan-liu-casia]

Hi @elladyr ,

I encountered an error while trying to make a submission​. During the "Push to evalai" step, the process failed with the following error, Could you please confirm if the submission is still closed?

Error: Sorry, cannot accept submissions since challenge phase is not public

Use evalai challenges to fetch the active challenges.

Use evalai challenge CHALLENGE phases to fetch the active phases.

Error: Process completed with exit code 1.

[elladyr]

Hi @siyuan-liu-casia, we are still working with EvalAI to fix some submission issues, hence the challenge remains as a private one. We will make an announcement once the challenge is up. Thank you for your patience.

[sharinka0715]

Hello MyoChallenge team,

May I inquire about the specific calculation formula and meaning of the 'effort' metric in the competition? As I understand it, 'effort' represents the total muscle activation. Therefore, a value closer to 0 would indicate that the policy consumes the least amount of force and is thus a better strategy.

However, the 'effort' value on the competition platform is currently negative, and it is still indicated that a lower value is better. Does this mean that a larger absolute value of 'effort' results in a higher ranking?

[cherylwang20]

Hi Jun,

Thanks for bringing this up. The effort metric reflects overall muscle activation — lower values mean the policy is using less force and is therefore more efficient (larger absolute value in effort means lower ranking). At the moment, the leaderboard is ranking this metric the wrong way, which has caused confusion. We’ll correct this in the next phase, but for now please use the score metric as the main reference for ranking.

Best,
Cheryl

[TaoshuaiZ]

Hello,

We are currently working on the second phase of the competition and have noticed that some of the ball trajectories appear unusual. They do not seem feasible for a robot to return, nor do they resemble those observed in real table tennis matches. I have recorded the trajectories in a video. Could you please take a look and let us know if it might be possible to make some adjustments to these trajectories?

Thank you sincerely!

test.mp4

[cherylwang20]

Hi,

Thank you for the question. We managed to push a hot fix. Can you install the latest Myosuite and let us know if the issues are solved?

Best,
Cheryl

[TaoshuaiZ]

Hello,

Thank you for your reply. I have checked the latest Myosuite and the issues are solved!

[sharinka0715]

Hello,

I would like to ask about the relationship between the Phase 2 online evaluation environment and the open-source code. The documentation mentions that the exact evaluation would be hidden, but it does not specify exactly what is being hidden. Regarding this, I have just one question:

In the Phase 2 online evaluation environment, does the randomness consist of, and only consist of, the mass of the paddle, the friction of the ping-pong ball, and the ball’s initial position and velocity, with no other random factors involved?

For this question, may I receive one of the following responses: yes, no, or unable to disclose? I look forward to your reply. Thank you very much!

[cherylwang20]

Hi Jun,

Thank you for your question. I may answer 'Yes' to this question.

In addition to all Table Tennis participants, we may be releasing an additional bonus round with special prizes. So stay tuned on that!

Best,
Cheryl

[Lee-Jun-Hyuk-37]

Dear MyoChallenge Team,

According to the official MyoChallenge 2025 website, Phase 2 is scheduled to end on October 31, and the results will be announced on December 2. However, on the EvalAI page, the competition is listed as ending on December 21. Could you please confirm which schedule is the correct one?

Additionally, could you clarify the benefits or opportunities provided to the winners? For example, up to which ranking are participants eligible for awards, and do winners have the chance to present at NeurIPS or publish a paper?

Best,
Jun Hyuk

[cherylwang20]

Hi Jun Hyuk,

Thank you for your question. Phase 2 is scheduled to end on Oct 31st, but we plan on extending it for another week due to high requests. The evalai page was set to a later date for internal evaluations. We apologize for the confusion.

This year's award will include, but might not be limited to

1. First place tennis
2. First place soccer
3. DEI Award
4. Student Award.

We might have additional ones based on the availability of funding.

All top 3 solutions in each track will be invited to co-author on a NeurIPS dataset & benchmark paper, and our last year's submission was quite successful. In addition, all six teams (if eligible) will receive swags and plate certificates via mail.

Hope this clarifies and let us know if yu have any additional questions.

Best ,
Cheryl

[sharinka0715]

Hello MyoChallenge Team,

I would like to inquire about the P2 test cases for the table tennis task. During the competition, I felt that the difficulty of the P2 test environment was significantly higher than the local evaluation. I would like to inquire: when designing the environment, did you ensure that every ball served is guaranteed to be returnable by the robot, or at least appears to be?

Alternatively, is the data generation method for the online evaluation environment simply a version of the P2 environment we have, but with a wider range of randomization, potentially including some balls that are impossible to return (for example, the trajectory is outside the myoarm's maximum reachable workspace, or the speed is too fast to react to)?

I look forward to your reply. Thank you!

[cherylwang20]

Hi Jun,

Thanks for the question. All balls are guaranteed to land on the participant's side of the table.

And yes, I confirm that the P2 online evaluation is just a wider range of randomization of the local parameter, and might potentially include trajectories that are very hard to catch.

Please let us know if you have any additional questions.

Best,
Cheryl

[omgxu]

Hello MyoChallenge Team,
In the soccer task, we noticed that the obs_dict does not include the goalkeeper position, preventing us from planning the kicking direction based on their location. We'd like to know if this is by design or an oversight.
Thank you.

[elladyr]

Hello Jinhao,

Thank you for pointing it out. Indeed, this was an oversight. A hotfix should be coming soon.
Stay tuned for updates.

Best,
Chun Kwang

[omgxu]

Hi Chun Kwang,

Thanks so much for the quick update on the goalkeeper position issue—really appreciate the team’s prompt response and the upcoming hotfix.

On a quick note about the task progress: we’re making steady headway, but a few internal adjustments have cropped up lately. We wanted to check if there’s a little flexibility with the timeline to make sure we can deliver quality work once the hotfix is in place.

No rush on your reply, and thanks again for your support!

[elladyr]

Hi Jinhao,

The hotfix has been released. The goalkeeper position can be accessed with the observation key "goalkeeper_pos", which provides the (X,Y) coordinates of the goal keeper.

Best,
Chun Kwang

[sharinka0715]

Hello MyoChallenge Team,

I am writing to inquire about the ranking criteria of the table tennis phase 2. I think the rule on the eval.ai website might be outdated, which is described as follows:

Ranking Criteria:

• Primary Criterion (Score): Teams must first achieve a table tennis task success rate of 90%* or higher to qualify for further ranking.
• Secondary Criterion (Effort): Among the teams that score within 10%* of each other, the ranking will be based on the muscle activation effort. In other words, it's important to use less (muscle-activation) effort to solve these tasks.

(*) These thresholds can be changed at the discretion of the organizers.

I think it is too challenging to achieve a 90% success rate in phase 2, so perhaps this threshold should be reconsidered. Also, the score gap of 10% seems to be too wide, and I am not sure whether it should be changed.

I am curious about when and how these two thresholds will be confirmed by the organizers. Will the thresholds be confirmed before the end of the submission stage (though today is the last day of phase 2)?

I am looking forward to your reply. Thanks!

[cherylwang20]

Hi Jun,

Thank you for the message. We would like to clarify that this is the criterion for final ranking, not for qualifying for ranking.
For example, if more than one team scored more than 90%, we will re-rank the participants again based on the secondary criterion (muscle activation).

If no participants score more than 90%, then the remaining teams that score within 3% (as opposed to 10%) of each other (referenced from the top team), will be re-ranked again based on the secondary criterion.

Hope this helps clarify your question.

-Cheryl