potential exposure of local path and identity of the user #249
Description
After many trials of prompt engineering testing against the same file, LangExtract started to extract the source name of the file, and hallucinate part of its content.
The output revealed the location of the file, but if it was a remote location, it may have exposed the source along with the open port, or, if the path is using the user identity name, they be spread out within the results without knowing it.
Hypothesis - the prompt contained instructions to extract the venue mentioned in the content of the source file, but instead the extractor extracted the source file name as a file location.
One could think there is a semantic misunderstanding. BUT, isn't source file name an external variable ?
how is it possible that the extractor extract the input name, and not the content of the input ?
Please propose some possible hypothesis.
Below the code and results:
# This is the line that will be exposed!
input_text = "/Users/[USERNAME]/[PATH]/9844_test_item.txt"
# Prompt and Examples:
# 1. Define the prompt and extraction rules
prompt = textwrap.dedent("""\
Constraints:
- Extract ONLY text spans that are explicitly written in the INPUT or that can be directly inferred from the wording (do not invent or generalize).
- Focus on values discussed as important.
- A value = either (1) a principle used as a premise to motivate a process, OR (2) a judgment about consequences of a process.
- If the article names the value (e.g., "Performance", "Efficiency"), use that exact name.
- If the article does not name a value but describes it (e.g., "lack of discussion of potential harms"), extract the phrase from the text and classify it as a value.
- Do not produce values, processes, or venues that are not explicitly mentioned in the INPUT.
Extraction schema:
1. Values: the name or phrase … attributes: must include [Premise] or [Consequence]
2. Processes: name of the process … attributes: inferred from text (e.g., data collection, data processing, benchmarking)
3. Venues: places where processes occur (physical or social, e.g., universities, conferences)
""")
# 2. Provide a high-quality example to guide the model
examples = [
lx.data.ExampleData(
text="Performance values are consistently operationalized as correctness averaged across predictions, without discussion of unequal impacts.",
extractions=[
lx.data.Extraction(
extraction_class="value",
extraction_text="Performance",
attributes={"type": "Premise", "description": "defined as correctness averaged across predictions"}
),
lx.data.Extraction(
extraction_class="value",
extraction_text="without discussion of unequal impacts",
attributes={"type": "Consequence"}
),
lx.data.Extraction(
extraction_class="process",
extraction_text="evaluating predictions",
attributes={"type": "data processing"}
),
]
),
lx.data.ExampleData(
text="It is extremely rare for papers to mention negative potential at all. This is striking for papers that advance identification of people in images, face-swapping, and video synthesis.",
extractions=[
lx.data.Extraction(
extraction_class="value",
extraction_text="rare mention of negative potential",
attributes={"type": "Consequence"}
),
lx.data.Extraction(
extraction_class="process",
extraction_text="identification of people in images",
attributes={"type": "data processing"}
),
lx.data.Extraction(
extraction_class="process",
extraction_text="face-swapping",
attributes={"type": "synthetic data"}
),
lx.data.Extraction(
extraction_class="process",
extraction_text="video synthesis",
attributes={"type": "synthetic data"}
),
]
),
lx.data.ExampleData(
text="The most frequently uplifted values are Performance, Generalization, Efficiency, Building on past work, and Novelty.",
extractions=[
lx.data.Extraction(
extraction_class="value",
extraction_text="Performance",
attributes={"type": "Premise"}
),
lx.data.Extraction(
extraction_class="value",
extraction_text="Generalization",
attributes={"type": "Premise"}
),
lx.data.Extraction(
extraction_class="value",
extraction_text="Efficiency",
attributes={"type": "Premise"}
),
lx.data.Extraction(
extraction_class="value",
extraction_text="Building on past work",
attributes={"type": "Premise"}
),
lx.data.Extraction(
extraction_class="value",
extraction_text="Novelty",
attributes={"type": "Premise"}
),
]
),
lx.data.ExampleData(
text="We analyzed the 100 most highly cited papers from NeurIPS and ICML, from the years 2008, 2009, 2018, and 2019.",
extractions=[
lx.data.Extraction(
extraction_class="venue",
extraction_text="NeurIPS",
attributes={"type": "conference"}
),
lx.data.Extraction(
extraction_class="venue",
extraction_text="ICML",
attributes={"type": "conference"}
),
]
),
]
# as in the example
result = lx.extract(
text_or_documents=input_text,
prompt_description=prompt,
examples=examples,
model_id="llama3.2", # Using Ollama on local computer
model_url="http://localhost:11434",
extraction_passes=3, # Multiple passes for improved recall
max_workers=20, # Parallel processing for speed
max_char_buffer=1000, # Smaller contexts for better accuracy
fence_output=False,
use_schema_constraints=False
)
Results I got:
{
"value_": {
"Datafication": {
"count": 1,
"attributes": [
{
"type": [
"Premise"
]
}
]
},
"Performance": {
"count": 2,
"attributes": [
{
"type": [
"Premise"
]
},
{
"type": [
"Premise"
]
}
]
},
"without discussion of unequal impacts": {
"count": 1,
"attributes": [
{
"type": [
"Consequence"
]
}
]
},
"Efficiency": {
"count": 1,
"attributes": [
{
"type": [
"Consequence"
]
}
]
}
},
"process_": {
"data processing": {
"count": 1,
"attributes": [
{
"type": [
"data collection"
]
}
]
}
},
"venue_": {
"/Users/[USERNAME]/[PATH]/9844_test_item.txt": {
"count": 1,
"attributes": [
{
"type": [
"file location"
]
}
]
}
}
}
NOTE:
I tried to communicate a security issue to Google on the link you provided, but it was a bit complicated for me, especially for structuring all the info. Not familiar and quite long process, hope it helped.