Apple silicon support

[Seanspt]

I'm trying to run MineStudio on Mac. Currently stuck on mcprec-6.13.jar. It is built with LWJGL 3.2.2, which needs to be >=3.3.0 to support apple silicon. Could a new jar be built with LWJGL upgraded?

[Seanspt]

Found the issue in #10 , I'll dig into that.

[Seanspt]

I got it work with minerl following the steps in this issue.
But python -m minestudio.simulator.entry shows the following error.

ERROR:minestudio.simulator.minerl.env._multiagent:Traceback (most recent call last):
  File "/Users/sean/projects/MineStudio/minestudio/simulator/minerl/env/_multiagent.py", line 335, in step
    reward, done, sent = struct.unpack("!dbb", reply)
TypeError: a bytes-like object is required, not 'NoneType'

I'll pick up from here later.

If someone is interested. Here are the steps I take to install minestudio on M4 mac.

• clone the project
• delete cuda-python dependency in pyproject.toml
• comment out cuda related content in gpu_utils.py
• add -XstartOnFirstThread and remove xvfb-run in launchClient.sh.
• pip install .
• replace the mcprec-6.13.jar

[Seanspt]

I realized that the code in mcprec-6.13.jar is modified to support minestudio features.
Could I get a copy or a diff patch of these code to rebuild the jar file?

[phython96]

I realized that the code in mcprec-6.13.jar is modified to support minestudio features. Is this part of code open sourced or could I get a copy to rebuild the jar file?

Thanks for your interest!! Here is the source code:
https://drive.google.com/file/d/1aS2eNTcxPq8E6UcK4iQaU1IIVtXkL40z/view?usp=sharing

[Seanspt]

I realized that the code in mcprec-6.13.jar is modified to support minestudio features. Is this part of code open sourced or could I get a copy to rebuild the jar file?

Thanks for your interest!! Here is the source code: https://drive.google.com/file/d/1aS2eNTcxPq8E6UcK4iQaU1IIVtXkL40z/view?usp=sharing

Thank you. I found this on MCP-Reborn page:

⚠️ WARNING ⚠️: You CANNOT publish any code generated by this tool.

Is it a law issue or what? The code is made to a patch in minerl project which might be related to this I think.

[phython96]

Sorry, I didn't get it. It seems that our work in Minecraft Simulator is just a wrapper for the MineRL work. What specifically do you think is inappropriate?

I realized that the code in mcprec-6.13.jar is modified to support minestudio features. Is this part of code open sourced or could I get a copy to rebuild the jar file?

Thanks for your interest!! Here is the source code: https://drive.google.com/file/d/1aS2eNTcxPq8E6UcK4iQaU1IIVtXkL40z/view?usp=sharing

Thank you. I found this on MCP-Reborn page:

⚠️ WARNING ⚠️: You CANNOT publish any code generated by this tool.

Is it a law issue or what? The code is made to a patch in minerl project which might be related to this I think.

[Seanspt]

Sorry, I didn't get it. It seems that our work in Minecraft Simulator is just a wrapper for the MineRL work. What specifically do you think is inappropriate?

I realized that the code in mcprec-6.13.jar is modified to support minestudio features. Is this part of code open sourced or could I get a copy to rebuild the jar file?

Thanks for your interest!! Here is the source code: https://drive.google.com/file/d/1aS2eNTcxPq8E6UcK4iQaU1IIVtXkL40z/view?usp=sharing

Thank you. I found this on MCP-Reborn page:

⚠️ WARNING ⚠️: You CANNOT publish any code generated by this tool.

Is it a law issue or what? The code is made to a patch in minerl project which might be related to this I think.

I guess the reason they give that warning is that the code MCP-Reborn generated is commercial code. So publishing it is not legitimate. Anyway just a guess, never mind.

[Seanspt]

I realized that the code in mcprec-6.13.jar is modified to support minestudio features. Is this part of code open sourced or could I get a copy to rebuild the jar file?

Thanks for your interest!! Here is the source code: https://drive.google.com/file/d/1aS2eNTcxPq8E6UcK4iQaU1IIVtXkL40z/view?usp=sharing

After rebuild the jar file. I get python -m minestudio.simulator.entry work on my mac. I also tried docker, which was much slower.
Docker result:

Speed Test Status: 
Average Time: 0.04 
Average FPS: 26.56 
Total Steps: 50 

Speed Test Status: 
Average Time: 0.03 
Average FPS: 28.97 
Total Steps: 100 

Mac result:

Speed Test Status: 
Average Time: 0.01 
Average FPS: 99.46 
Total Steps: 100 

I'll try more examples later.

• Simulator
  • Graphical User Interfaces
• Inference
  • Tutorial: Inference with VPT
  • GROOT
  • STEVE-1
• offline training
  • Tutorial: Fine-tuning VPT to a Hunter
• online training
• bechmark

[muzhancun]

Thank you. I found this on MCP-Reborn page:

⚠️ WARNING ⚠️: You CANNOT publish any code generated by this tool.

Is it a law issue or what? The code is made to a patch in minerl project which might be related to this I think.

Here is an answer related to your question. And thanks for your excellent test on MacOS!

[Seanspt]

When I run Tutorial: Inference with VPT,
two kinds of error may happen by chance. Some investigation need to be done later.

ray::Worker.get_next() (pid=13553, ip=127.0.0.1, actor_id=49a359c6982fc641ad6d01f101000000, repr=<minestudio.inference.generator.mine_generator.Worker object at 0x10c7555d0>)
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 339, in sample
    return OrderedDict([
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 340, in <listcomp>
    (k, v.sample(bs)) for k, v in self.spaces.items()
TypeError: Text.sample() takes 1 positional argument but 2 were given

During handling of the above exception, another exception occurred:

ray::Worker.get_next() (pid=13553, ip=127.0.0.1, actor_id=49a359c6982fc641ad6d01f101000000, repr=<minestudio.inference.generator.mine_generator.Worker object at 0x10c7555d0>)
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 339, in sample
    return OrderedDict([
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 340, in <listcomp>
    (k, v.sample(bs)) for k, v in self.spaces.items()
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 343, in sample
    return OrderedDict([
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 344, in <listcomp>
    (k, v.sample()) for k, v in self.spaces.items()
  File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 497, in sample
    return np.array(np.reshape(strings, self.shape), np.dtype)
TypeError: Cannot convert np.dtype into a dtype.

[04:41:27] [EnvServerSocketHandler/INFO]: Received Mission token 9b1fdae8-f574-497c-8e9b-1e7d5afb2fd8:0:0:1:true:0
[04:41:27] [EnvServerSocketHandler/INFO]: Received mission init command  <MissionInit xmlns="http://ProjectMalmo.microsoft.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" SchemaVersion="" PlatformVersion="0.37.0"><Mission><About><Summary>MineRLHumanSurvival-v0</Summary></About><ModSettings><MsPerTick>50</MsPerTick></ModSettings><ServerSection><ServerInitialConditions><Time><AllowPassageOfTime>true</AllowPassageOfTime></Time><AllowSpawning>true</AllowSpawning></ServerInitialConditions><ServerHandlers><DefaultWorldGenerator forceReset="true" generatorOptions="{}"/><ServerQuitWhenAnyAgentFinishes/></ServerHandlers></ServerSection><AgentSection mode="Survival"><Name>MineRLAgent0</Name><AgentStart><LowLevelInputs>true</LowLevelInputs><GuiScale>1.0</GuiScale><GammaSetting>2.0</GammaSetting><FOVSetting>70.0</FOVSetting><FakeCursorSize>16</FakeCursorSize><Inventory/></AgentStart><AgentHandlers><FileBasedPerformanceProducer/><PauseCommand/><VideoProducer want_depth="false"><Width>640</Width><Height>360</Height></VideoProducer><ObservationFromFullInventory flat="false"/><ObservationFromEquippedItem/><ObservationFromFullStats/><ObservationFromFullStats/><HumanLevelCommands/><CameraCommands/><ChatCommands/><IsGuiOpen/></AgentHandlers></AgentSection></Mission><ExperimentUID>9b1fdae8-f574-497c-8e9b-1e7d5afb2fd8</ExperimentUID><ClientRole>0</ClientRole><ClientAgentConnection><ClientIPAddress>127.0.0.1</ClientIPAddress><ClientMissionControlPort>0</ClientMissionControlPort><ClientCommandsPort>0</ClientCommandsPort><AgentIPAddress>127.0.0.1</AgentIPAddress><AgentMissionControlPort>0</AgentMissionControlPort><AgentVideoPort>0</AgentVideoPort><AgentDepthPort>0</AgentDepthPort><AgentLuminancePort>0</AgentLuminancePort><AgentObservationsPort>0</AgentObservationsPort><AgentRewardsPort>0</AgentRewardsPort><AgentColourMapPort>0</AgentColourMapPort></ClientAgentConnection></MissionInit>
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000035007c158, pid=78988, tid=0x0000000000015207
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.78.0.19-CA-macos-aarch64) (8.0_412-b08) (build 1.8.0_412-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.412-b08 mixed mode bsd-aarch64 compressed oops)
# Problematic frame:
# C  [liblwjgl_stb.dylib+0x4c158]
[error occurred during error reporting (printing problematic frame), id 0xe0000000]

# Core dump written. Default location: /cores/core or core.78988
#
# An error report file with more information is saved as:
# /Volumes/Mac Studio/projects/minerl-demo/tmp/stark_tech_instances/d9d21baa-1d57-4c46-af98-c1c4d68578a4/hs_err_pid78988.log
[04:41:27] [EnvServerSocketHandler/INFO]: [STDOUT]: Setting width, height to 640, 360
[04:41:27] [EnvServerSocketHandler/INFO]: [STDOUT]: Gamma: 2.0
[04:41:27] [EnvServerSocketHandler/INFO]: [STDOUT]: FOV: 70.0
[04:41:27] [EnvServerSocketHandler/INFO]: [STDOUT]: GuiScale: 1.0
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/env/launchClient.sh: line 47: 78988 Abort trap: 6           java -Xmx$maxMem -XstartOnFirstThread -jar $fatjar --envPort=$port --envSeed=$seed

[Seanspt]

I also did Tutorial: Fine-tuning VPT to a Hunter, whose code is out of date BTW.
MPS backend doesn't support float64, so I changed the data type to float32 in dataset to_tensor as followed

def to_tensor(self, item: Union[np.ndarray, List, Dict]) -> Union[torch.Tensor, List, Dict]:
        """Recursively converts NumPy arrays within a nested structure (list or dict) to PyTorch tensors.

        If the input item is a NumPy array, it's converted to a PyTorch tensor.
        If it's a list, the conversion is applied to each element.
        If it's a dictionary, the conversion is applied to each value.
        Other types are returned as is.

        :param item: The item to convert. Can be a NumPy array, or a list/dict containing them.
        :type item: Union[np.ndarray, List, Dict]
        :returns: The item with NumPy arrays replaced by PyTorch tensors.
        :rtype: Union[torch.Tensor, List, Dict]
        """
        # Convert numpy array to torch tensor.
        if isinstance(item, np.ndarray):
            tensor = torch.from_numpy(item)
            if tensor.dtype == torch.float64:
                tensor = tensor.float()
            return tensor
        elif isinstance(item, List):
            return [self.to_tensor(val) for val in item]
        elif isinstance(item, Dict):
            return {key: self.to_tensor(val) for key, val in item.items()}
        else:
            return item

If float64 is not crucial for then performence, can I leave the code like this and make it a default behavior for all platforms?

[Seanspt]

It seems promising for apple silicon mac to run minestudio. So I created a pr, and uploaded the jar file, in case some is also interested.

[muzhancun]

ray::Worker.get_next() (pid=13553, ip=127.0.0.1, actor_id=49a359c6982fc641ad6d01f101000000, repr=<minestudio.inference.generator.mine_generator.Worker object at 0x10c7555d0>)
File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 339, in sample
return OrderedDict([
File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 340, in
(k, v.sample(bs)) for k, v in self.spaces.items()
File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 343, in sample
return OrderedDict([
File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 344, in
(k, v.sample()) for k, v in self.spaces.items()
File "/Users/sean/datadisk/miniconda3/envs/minestudio/lib/python3.10/site-packages/minestudio/simulator/minerl/herobraine/hero/spaces.py", line 497, in sample
return np.array(np.reshape(strings, self.shape), np.dtype)
TypeError: Cannot convert np.dtype into a dtype.

This issue occurs when the environment doesn't start properly, which can happen occasionally. Fortunately, it doesn't affect online training because crash protection is in place.

As for the second error, I haven't encountered it before—it's possible that it's specific to macOS.