Python Import issue undefined symbol: http_open_hdrs

[paulmac57]

Description: Ubuntu 18.04.4 LTS

i have followed the installation procedure and the dependencies installation procedures.
The test fails with ImportError: No module named _pybgpstream_caida

if I change the import line to "import pybgpstream as _pybgpstream" I then get an error which states ImportError: /usr/lib/x86_64-linux-gnu/libbgpstream.so.3: undefined symbol: http_open_hdrs

Also try from python interpreter below.

Python 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pybgpstream
Traceback (most recent call last):
File "", line 1, in
File "/home/initiate/.local/lib/python2.7/site-packages/pybgpstream/init.py", line 27, in
from .pybgpstream import *
File "/home/initiate/.local/lib/python2.7/site-packages/pybgpstream/pybgpstream.py", line 29, in
import _pybgpstream
ImportError: /usr/lib/x86_64-linux-gnu/libbgpstream.so.3: undefined symbol: http_open_hdrs

[bkbristlin]

I've just done a new install of Ubuntu 18.04.4 LTS and am experiencing the same issue. Mine is a fresh install and I encountered the issue running the test print script. Did this ever get resolved or anyone have any ideas?

burrito@tacosplease-ubs18-bgps1:~$ curl
https://raw.githubusercontent.com/CAIDA/pybgpstream/master/examples/tutorial_print.py | python3

% Total % Received % Xferd Average Dload Speed Upload Time Total Time Spent Time Left Current Speed
100 2252 100 2252 0 0 8309 0 --:--:-- --:--:-- --:--:-- 8340
Traceback (most recent call last):
File "", line 28, in
File "/usr/lib/python3/dist-packages/pybgpstream/init.py", line 27, in
from .pybgpstream import *
File "/usr/lib/python3/dist-packages/pybgpstream/pybgpstream.py", line 31, in
import _pybgpstream
ImportError: /usr/lib/x86_64-linux-gnu/libbgpstream.so.3: undefined symbol: http_open_hdrs
burrito@tacosplease-ubs18-bgps1:~$

[alistairking]

My best guess is that the version of libwandio that you have installed is too old -- that function wasn't exported until 4.2.0.
What process did you use to install bgpstream?

[bkbristlin]

I went through the process listed out on the site at the two links below. Might better if I build a new VM and start over. I had hit a couple issues with dependencies along the way that I didn't document.

https://bgpstream.caida.org/docs/install/bgpstream
https://bgpstream.caida.org/docs/install/pybgpstream

[bkbristlin]

I am up and running now with a fresh install. It turns out that I missed an error on the bgpstream install. I detailed the process I went through should someone come across this.

sudo apt update
sudo apt upgrade

This took me to Ubuntu 18.04.5 LTS

burrito@tacosplease-ubs18-bgps1:~$ sudo apt-get install bgpstream
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
bgpstream : Depends: bgpstream2-tools (= 2.0.0) but it is not going to be installed
Depends: libbgpstream2-dev (= 2.0.0) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Lots of google to figure this out. Found someone that mentioned using aptitude to install a package to better determine the dependency issues.

sudo apt-get install aptitude

burrito@tacosplease-ubs18-bgps1:~$ sudo aptitude install bgpstream
The following NEW packages will be installed:
bgpstream bgpstream2-tools{a} libbgpstream2{ab} libbgpstream2-dev{a} libwandio1{a}
0 packages upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 231 kB of archives. After unpacking 1,237 kB will be used.
The following packages have unmet dependencies:
libbgpstream2 : Depends: librdkafka-dev (>= 0.11.3) which is a virtual package and is not provided by any available package

Depends: librdkafka1 (>= 0.11.3) which is a virtual package and is not provided by any available package

The following actions will resolve these dependencies:

Keep the following packages at their current version:
1) bgpstream [Not Installed]
2) bgpstream2-tools [Not Installed]
3) libbgpstream2 [Not Installed]
4) libbgpstream2-dev [Not Installed]

Accept this solution? [Y/n/q/?] Y
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.

I searched everywhere and tried many methods. No installation candidates could be found for librdkafka-dev & librdkafka1. I checked the Ubuntu packaged page and it referenced the package being in "Universe"
https://packages.ubuntu.com/source/bionic/librdkafka

I checked and Universe appears to be already set even though things were failing.

burrito@tacosplease-ubs18-bgps1:/etc/apt$ tail sources.list
deb http://archive.ubuntu.com/ubuntu bionic main universe
deb http://archive.ubuntu.com/ubuntu bionic-security main universe
deb http://archive.ubuntu.com/ubuntu bionic-updates main universe

I added Universe anyways just to see.

sudo add-apt-repository universe
sudo apt update

burrito@tacosplease-ubs18-bgps1:/etc/apt$ sudo apt install librdkafka-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
librdkafka++1 librdkafka1
The following NEW packages will be installed:
librdkafka++1 librdkafka-dev librdkafka1
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 712 kB of archives.
After this operation, 3,115 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

This also installed librdkafka1.

burrito@tacosplease-ubs18-bgps1:/etc/apt$ sudo apt install librdkafka1
Reading package lists... Done
Building dependency tree
Reading state information... Done
librdkafka1 is already the newest version (0.11.3-1build1).
librdkafka1 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

After this I was able to install bgpstream

burrito@tacosplease-ubs18-bgps1:/etc/apt$ sudo aptitude install bgpstream
The following NEW packages will be installed:
bgpstream bgpstream2-tools{a} libbgpstream2{a} libbgpstream2-dev{a} libwandio1{a}
0 packages upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 231 kB of archives. After unpacking 1,237 kB will be used.
Do you want to continue? [Y/n/?] Y

Then verified it was installed

burrito@tacosplease-ubs18-bgps1:/etc/apt$ dpkg -l bgpstream
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==================================-======================-======================-==========================================================================
ii bgpstream 2.0.0 amd64 Meta package for full libbgpstream install

From there I was able to successfully install Python bgpstream

burrito@tacosplease-ubs18-bgps1:/etc/apt$ sudo apt-get install python3-pybgpstream
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
python3-dateutil
The following NEW packages will be installed:
python3-dateutil python3-pybgpstream
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 64.2 kB of archives.
After this operation, 323 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

And from there I was able to run the test command (note the switch to python3 from python)

burrito@tacosplease-ubs18-bgps1:/etc/apt$ curl https://raw.githubusercontent.com/CAIDA/pybgpstream/master/examples/tutorial_print.py | python3
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2252 100 2252 0 0 31718 0 --:--:-- --:--:-- --:--:-- 31718
valid ris.rrc06 1427846570.0
W 202.249.2.185 25152 W {'prefix': '144.104.37.0/24'}
valid ris.rrc06 1427846573.0
A 2001:200:0:fe00::6249:0 25152 A {'next-hop': '2001:200:0:fe00::9c1:0', 'as-path': '25152 2497 6939 47541 28709','communities': set(), 'prefix': '2a00:bdc0:e004::/48'}`

Lots of output followed, omitted for brevity.

[digizeph]

@bkbristlin Thank you for posting your solution! This will be very helpful for people who encountered the similar problems.

We do have librdkafka in our environment for some reason, that's why this problem did not show up for us. Below is the search result from our test environment:

limbo@debian-pkg:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic

limbo@debian-pkg:~$ apt search librdkafka
Sorting... Done
Full Text Search... Done
golang-github-confluentinc-confluent-kafka-go-dev/bionic,bionic 0.11.0-2 all
  Apache Kafka Golang client by Confluent

librdkafka++1/bionic 0.11.3-1build1 amd64
  library implementing the Apache Kafka protocol (C++ bindings)

librdkafka-dev/bionic 0.11.3-1build1 amd64
  library implementing the Apache Kafka protocol (development headers)

librdkafka1/bionic 0.11.3-1build1 amd64
  library implementing the Apache Kafka protocol

python-confluent-kafka/bionic 0.11.0-1 amd64
  Python client to interact with Kafka - Python 2.7

python3-confluent-kafka/bionic 0.11.0-1 amd64
  Python client to interact with Kafka - Python 3.x