No support for XMILE models with multiple array dimensions?

[Steve-Yang-Sys]

Hello,

Recently, I tried to read a Stella model with PySD package, but it does not work at all.

After several tests, I found that PySD cannot read arrayed variables with multiple (>1) dimensions or subscripts in Stella. Additionally, it also does not support arrayed variables indexed by Number (NOT Label). The sample error message is given as below:

---

KeyError Traceback (most recent call last)
in
----> 1 model = pysd.read_xmile(MODEL_PATH)

~/opt/anaconda3/lib/python3.8/site-packages/pysd/pysd.py in read_xmile(xmile_file, data_files, initialize, missing_values)
85
86 # build Python file
---> 87 py_model_file = ModelBuilder(abs_model).build_model()
88
89 # load Python file

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/python_model_builder.py in init(self, abstract_model)
39 self.dict = abstract_model.dict.copy()
40 # load sections
---> 41 self.sections = [
42 SectionBuilder(section)
43 for section in abstract_model.sections

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/python_model_builder.py in (.0)
40 # load sections
41 self.sections = [
---> 42 SectionBuilder(section)
43 for section in abstract_model.sections
44 ]

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/python_model_builder.py in init(self, abstract_section)
87 abstract_section.subscripts, self.root)
88 # Load the elements in the section
---> 89 self.elements = [
90 ElementBuilder(element, self)
91 for element in abstract_section.elements

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/python_model_builder.py in (.0)
88 # Load the elements in the section
89 self.elements = [
---> 90 ElementBuilder(element, self)
91 for element in abstract_section.elements
92 ]

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/python_model_builder.py in init(self, abstract_element, section)
548 self.section = section
549 # Get the subscripts of the element after merging all the components
--> 550 self.subscripts = section.subscripts.make_merge_list(
551 [component.subscripts[0] for component in self.components])
552 # Get the subscript dictionary of the element

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/subscripts.py in make_merge_list(self, subs_list, element)
296 """
297 coords_set = [set() for i in range(len(subs_list[0]))]
--> 298 coords_list = [
299 self.make_coord_dict(subs)
300 for subs in subs_list

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/subscripts.py in (.0)
297 coords_set = [set() for i in range(len(subs_list[0]))]
298 coords_list = [
--> 299 self.make_coord_dict(subs)
300 for subs in subs_list
301 ]

~/opt/anaconda3/lib/python3.8/site-packages/pysd/builders/python/subscripts.py in make_coord_dict(self, subs)
257 coordinates[sub] = self.subscripts[sub[:-1]]
258 else:
--> 259 coordinates[sub] = self.subscripts[sub]
260 return coordinates
261

KeyError: '1'

---

Normally, we try to translate large and complex SD models into Python environment for deeper analysis, but this support limitation really hinders the impact of PySD package. Hopefully, this issue is something under your kind consideration.

[enekomartinmartinez]

Thanks @Steve-Yang-Sys

Could you please share your model or a simpler version of it that reproduces this error?

The issue may be with the subscripts parsing, in this case, I may be able to solve it easily.

[Steve-Yang-Sys]

Thanks for your kind reply very much @enekomartinmartinez

Sure. I am happy to share my test files with you. In the sample test models, I found that PySD package cannot interpret Stella models with arrayed variables at least in three cases:

1. If the arrayed variable is in "non-a2a-graphical function" (test 1), it will return "IndexError: list index out of range";
2. If the arrayed variable is indexed by Number (test 2), it will return "KeyError: '1'";
3. If the arrayed variable is used with more than one dimensions (test 3), it will also return "KeyError message".

I tried to look at the "subscripts.py" file to detect the problems by myself, but I have no idea how the subscripts have been translated in PySD. If you can help fix this parsing issue, that would really do me a big favor 😊

Test models.zip

[enekomartinmartinez]

Thank you for the examples @Steve-Yang-Sys :)

I will take a look at the bug this week!

[enekomartinmartinez]

What I found in a quick look is :

Bug in non-a2a-gf_test_1.stmx:

• xmile parsers eqn node to be defined inside the element level, not in the previous level, if defined in the previous level we should take it from it (need to modify a little how the parser works).

Bug in non-a2a_test_2.stmx:

• Need to add support to dimensions defined with "size" <dim name="Dimension_1" size="2"/> (it should be easy to solve)

Bug in non-a2a_test_3.stmx:

• xmile parser doesn't remove whitespace after the colon in the element subscripts (easy to solve with a 'strip')

[enekomartinmartinez]

Hi @Steve-Yang-Sys

The three tests models are running with the version in https://github.com/SDXorg/pysd/tree/dev_xmile

Could you please test your model with this branch? And let me know if it runs and gives the same result?

Also, could I use the tests models you share for our testing suite?

Cheers!

[Steve-Yang-Sys]

Hi @enekomartinmartinez,

Sorry for my late reply because I was out of office for the last few weeks.

Thanks for your great job so much 👍. After simple tests with previous sample models, I think PySD can now support arrayed variables in xmile model file much better than before.

However, when testing with my own model, I got a new error message as follow:

File "/Users/yang/opt/anaconda3/envs/Calibration1/lib/python3.13/site-packages/pysd/pysd.py", line 80, in read_xmile
xmile_file_obj.parse()
~~~~~~~~~~~~~~~~~~~~^^
File "/Users/yang/opt/anaconda3/envs/Calibration1/lib/python3.13/site-packages/pysd/translators/xmile/xmile_file.py", line 105, in parse
section.parse()
~~~~~~~~~~~~~^^
File "/Users/yang/opt/anaconda3/envs/Calibration1/lib/python3.13/site-packages/pysd/translators/xmile/xmile_section.py", line 121, in parse
[component.parse(self.behaviors) for component in self.components]
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/Users/yang/opt/anaconda3/envs/Calibration1/lib/python3.13/site-packages/pysd/translators/xmile/xmile_element.py", line 194, in parse
self._parse_component(subnode, behaviors, asts=asts)[0])
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Flow._parse_component() got an unexpected keyword argument 'asts'

Moreover, it is found that the current version does not support any functions (like SUM, MEAN, MIN, MAX, RANK, SIZE etc.) used for arrayed variables in Stella models, as demonstrated in the new test model. Personally, I think adding array functions in PySD is straightforward and can be easily accomplished with numpy.sum(), numpy.mean(), numpy.max(), numpy.min().

Sure. All the test models are free to use or distribute, and I fully support the development of PySD community.

Cheers :)

non-a2a (array functions).stmx.zip

[enekomartinmartinez]

Hi @Steve-Yang-Sys

I have fixed the error and added support for mean and stddev, other functions were already supported.

Could you test your model again? You may need to do a force pull, sorry!

[Steve-Yang-Sys]

Hi @Steve-Yang-Sys

I have fixed the error and added support for mean and stddev, other functions were already supported.

Could you test your model again? You may need to do a force pull, sorry!

Good job, indeed! @enekomartinmartinez

Thanks for adding support of array functions very much. After further tests, I found that there would be error messages at least in these two test scenarios:

1. If we add comments ([]) for model variables, PySD will raise "IncompleteParseError(text, node.end, self)", see the attached test model (array with comments (test 1).stmx);
2. Currently, PySD can translate arrayed auxiliary variables successfully, but when dealing with arrayed stocks and flows, the "AttributeError: 'NoneType' object has no attribute 'strip'" will show up, see the test model (array SFD (test 2).stmx).

Can you fix these bugs? Really appreciate :)

If there are any questions, please feel free to let me know.

Array test models.zip

[enekomartinmartinez]

Hi, sorry for the late response. I was on holiday with no access to a computer, and I'm currently rather busy with my PhD.

I will take a look at the bugs as soon as I have some free time. Probably by the end of the month. Sorry! :S

Also, if you or anyone you know would be able to help with maintaining the library, that would also be welcome, as I am currently the only active developer and I don't really have much free time. :)