Config DATA api for local ripple network

[treisto]

Hi all,

Anybody could give any insights into how to set up the configs for DATA.api to connect to a local cluster of 3 rippled validators, please?

Validators' local IPs and public keys:
"nHB2dJDdSCmYtNSavvbG7Bsc46zFXWuvH7FvFqp9Nma1EQpRpGbz" : "192.168.1.97",
"nHBZM8xDhz2gQ53jkQWbHseALzANHPED8KYavrXKQM85EMXrbpU8" : "192.168.1.98",
"nHBWGxXEfRe2qXg8gFfemMo7vEmGkSJ4LD6xq6pU9VctwV6cpQSK" : "192.168.1.132"

"Data.api" is running on a 4th machine, inside VirtualBox-Ubuntu 18, in the same local network, IP 192.168.1....

Data.api works locally (returns health ok at 127.0.0.1:7111/v2/health), I can't connect it to the validator to extract the history.
When I enter "node backfill.js", I get:
NotConnectedError: connect ECONNREFUSED 192.168.1.97:443

Below are the best settings that I could come up with:

{
"port": 7111,
"logLevel": 2,
"logFile": "log_test",
"genesis_ledger": 1,
"hbase" : {
"logLevel" : 2,
"prefix" : "test_",
"servers" : [
{"host" : "//127.0.0.1", "port": 9090} OK WORKS
]
},
"ripple": {
"server" : "ws://192.168.1.97:443"
},
"cacheControl": {
"max-age": 10,
"stale-while-revalidate": 30,
"stale-if-error": 300
},
"hbase-rest" : {
"prefix" : "test_",
"host" : "127.0.0.1",
"port" : 20550 OK WORKS
},
"validators-config" : "validators.config.json.example",
"fee_url": "http://192.168.1.97:51234",
"hdfs": { DO I NEED THIS for local Hbase?
"user": "impala",
"host": "localhost",
"port": 50070,
"path": "/path/"
},
"recipients": ["user@example.com"], Is this for hdfs? DO I NEED THIS for local Hbase?
"rippleds_url": "https://data.ripple.com/v2/network/topology/nodes", Don't know how to serve this? What can I put here?
"rateLimit": { What are these for?
"max": 60,
"duration": 60000,
"whitelist": [],
"blacklist": []
},
"redis": { DO I NEED THIS for local Hbase?
"host": "localhost",
"port": "6379"
}
}

================VALIDATORS=================

{
"validator-domains" : {
"nHB2dJDdSCmYtNSavvbùuuuuuuuuuuuuvH7FvFqp9Nma1EQpRpGbz" : "192.168.1.97",
kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkNHPED8KYavrXKQM85EMXrbpU8" : "192.168.1.98",
"nHBWGxXEfRe2qXg8gFfemMo7vEmGkSJ4LD6xq6pU9VctwV6cpQSK" : "192.168.1.132"
},
"unverified-validators" : [
"nHBZM8xDhz2gQ5kkkkkkkkkkkkklllNHPED8KYavrXKQM85EMXrbpU8",
"nHBWGxXEfRe2qXg8gFfjjjjjjjjjjjjjjjjjjjjjjjjjjjjJ4LD6xq6pU9VctwV6cpQSK",
"nHB2dJDdSCmYtfffffffffffffffffff46zFXWuvH7FvFqp9Nma1EQpRpGbz"
]
}

Thank you very much!

[treisto]

@shekenahglory Hi Shekenahglory, could you give me any hints if possible, please?
How should I set the IPs/ports on the validator(s) cfgs, and how should I set data.api cfgs?
I don't need all the bells and whistles, for now I would be happy if I can fetch the ledgers db - accounts, transactions..
Thank you very much.

Below is the cfg for the one validator I am trying to connect to (I run a cluster of 3 validators):

[server]
port_rpc_admin_local
port_peer
port_ws_admin_local
port_ws_public
port_public
#ssl_key = /etc/ssl/private/server.key
#ssl_cert = /etc/ssl/certs/server.crt

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer

[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws

[port_ws_public]
port = 6005
ip = 127.0.0.1
protocol = wss

[port_public]
ip = 0.0.0.0
port = 443
protocol = ws

[ssl_verify]
0

[ips_fixed]
192.168.1.97 51235
192.168.1.132 51235

[peer_private]
1

[node_seed]
shEm9dCCCCCCC9XsXYXKrPmqft

[cluster_nodes]
n9LPJFoTLxCCCCCCCCCCCCCCwC3aLAYGhFcNNR61fD9DTc2w5L ripdbg1
n9KUMms9CCCCCCCCCCCCCCCEWy5Ghk3qj53aCPAbJRur2sTqwp ripdbg3

[validator-token]
eyJtYW5pZmVzdCI6IkpBQUFBQUZ4SWUwYkVlUVp1bGNsKzRadk44cGhXUWJNNWhlV3RKY0hN
YUVKcUpadWVRWm9jMMMMMMMMMMMMMMMMMMMMMMV1UnFDd2Q3U2JkU3hPTnJq
TXlsNWlka2N3UlFJaEFKc3IzL3g2U0RiRGprOHc0Mks2eU91M1FPbW4vNjVIeTM4bkxjbnJa
c1ROQWlBSnRlRTRpdjVqSjRJMytvS0VseEFjTmFUL3VoQnRlSVFyK29RdmVoemJESEFTUU53
RnpLN21kV3lUaTZMMMMMMMMMMMMMMMMMMMMMMMM9BK3FKZWZLMzkw
Smx3aE5ydGVLL09LWS8rQldDUHo0ejQ4VXptaHd3PSIsInZhbGlkYXRpb25fc2VjcmV0X2tl
eSI6IkJGMTcyRjJBMzNGQTZDOTdBQ0JBODhBNTA0NThGQzZFRURENzBCNjEwMzdEMjcwNjgz
RTQ3MzRBNUY2OURGRkMifQ==

[shekenahglory]

I do not recommend trying to set up your own data API, you need to be running hbase + an ingestion service that runs on storm in order to populate it with data. Then the API connects to that hbase cluster, not rippled directly. what is it you are trying to accomplish?

[treisto]

@shekenahglory Hello and thank you very much for your help.

I am running a local independent network not connected to ripple, 3 validators with full history.
All I need is to access in data.api the info related to the ledgers - accounts, transactions... The rest would be a "bonus".

I have deployed hbase, is connected to Data.api, I created the tables with the script, started Thrift with frame mod, and the REST server. Hbase seems to communicate ok with data api, I ran the "docker-compose" test and the tables pass it.

Now I need to populate hbase with data. For this, my understanding is that I have to run "import /live" to fetch the history real time as it is being written, or "import /backfill".
Q: How do these 2 commands relate to storm and hbase, please?

I found the storm code in your repo, thanks a lot for the hint.
Q: The data path would be Validator->Storm->Hbase->Data.api, is this correct?

Thank you very much for your time

[shekenahglory]

You can query that data directly from your validators though. The data API gives the same info, though it is more REST-like than rippled's apis. Is that what you are looking for? The version of the storm import used here is not exactly the same as what we are using in production, it should work though you may run into issues. The script to launch the storm topology is here https://github.com/ripple/rippled-historical-database/blob/master/storm/production/importer.sh

[treisto]

Thank you very much your help, I really appreciate it.
At some point the historical data might be used for machine learning applications or stats, and I had the feeling it would be a friendlier/better source for these purposes. Please tell me if you disagree, it would be really helpful.

[h6dworker]

"validators-config" : "validators.config.json.example",
"fee_url": "http://s1.ripple.com:51234",
"hdfs": {
"user": "impala",
"host": "localhost",
"port": 50070,
"path": "/path/"
},
"recipients": ["user@example.com"],
"rippleds_url": "https://data.ripple.com/v2/network/topology/nodes",
"rateLimit": {
"max": 60,
"duration": 60000,
"whitelist": [],
"blacklist": []
},
"redis": {
"host": "localhost",
"port": "6379"
}

No steps
(It was possible until the previous version)

Please tell me the correct procedure.