First of all, I'm very pleased that [all of my pull requests for devportal](https://github.com/steemit/devportal/pulls?q=is%3Apr+is%3Aclosed+author%3Ainertia186+review%3Aapproved) have reached the Steemit, Inc. official site, as [reported yesterday](https://steemit.com/steem/@steemitblog/new-year-new-devportal-updates). My goal was to try to keep my fork and the origin fork as 1-to-1 as possible. There's no reason for me to have unique documentation on my fork. It looks like it took them only a couple days to fully merge my changes that spanned over a month. Great job, Steemit, Inc! I believe moving forward, this should be much easier now that the relative URL structure has been addressed. Here, I'd like to go over the various `rake` tasks that have also been merged to the official repository. Some of these tasks are shown in the @steemitblog post from yesterday. But here's more detail on what they're all about. #### Repository https://github.com/inertia186/devportal Rake (`rake`) is a command-line tool that means "Ruby Make." The command implements tasks defined in a `Rakefile`, usually in a cloned git project. --- In the devportal Steem documentation project, the rake tasks are as follows: ```bash $ rake -vT rake ops_dump[vops,appbase] # Dump all operation types rake production:deploy # Deploy current master to GH Pages rake production:rollback # Rollback GH Pages rake scrape:api_defs # Scrape API Definitions rake scrape:javascript # Scrape steemjs docs rake scrape:python # Scrape pysteem docs rake scrape:tutorials # Scrape all known tutorial repositories rake scrape:tutorials:js[num,force] # Scrape JS-Tutorials rake scrape:tutorials:py[num,force] # Scrape PY-Tutorials rake scrape:tutorials:rb[num,force] # Scrape RB-Tutorials rake test:curl[apis] # Tests the curl examples of api definitions rake test:proof # Want some work to do? Run this report and get busy ``` I use many of these tasks to help with documentation. Here's a rundown of the most useful ones ... ### Scrape API Definitions For example, `rake scrape:api_defs` is especially useful. As it says, it scrapes the API. But what does that even mean? When it scrapes the API, it's asking the (full) node what methods are available and what those method signatures are. Then, it saves the scraped information to a `.yml` file so that jekyll can make pages to browse. For example, [`rc_api`](https://inertia186.github.io/devportal/apidefinitions/rc-api) is based on scraped data saved to [`rc_api.yml`](https://github.com/inertia186/devportal/blob/master/_data/apidefinitions/rc_api.yml). I don't *just* scrape the methods, though. Once scraped, I add details. Also, sometimes the scrape is missing some of them method signatures, so I add those too. You might think this is only useful for catching up to methods that already exist, but it's also possible to use it for upcoming methods. All I need is to point the task at a node running on a future release. Often, new methods show up on `api.steemitdev.com`, so this will scrape from there: ```bash TEST_NODE=https://api.steemitdev.com bundle exec rake scrape:api_defs ``` In fact, I could scrape from a testnet just as easily: ```bash TEST_NODE=https://testnet.steemitdev.com bundle exec rake scrape:api_defs ``` It also works just as well to run a [`tintoy`](https://steemit.com/testnet/@inertia/tin-toy) testnet if I'm looking for bleeding edge new methods: ```bash docker run -d -p 8090:8090 inertia/tintoy:latest # need to wait for it to fully deploy sleep 300 TEST_NODE=http://localhost:8090 bundle exec rake scrape:api_defs ``` And here's the result when I ran it just now: ``` Definitions for: account_by_key_api, methods: 1 Definitions for: account_history_api, methods: 4 Adding: enum_virtual_ops Changed: get_account_history Changed: get_ops_in_block Changed: get_transaction 4 methods added/updated in account_history_api Definitions for: block_api, methods: 2 Definitions for: condenser_api, methods: 84 Dropped method: get_account_bandwidth (recommend removal from _data/apidefinitions/condenser_api.yml) Definitions for: database_api, methods: 52 Changed: find_accounts Changed: find_comments Changed: find_escrows Adding: find_smt_token_emissions Adding: find_smt_tokens Changed: find_vesting_delegation_expirations Changed: find_votes Changed: find_witnesses Changed: get_active_witnesses Changed: get_config Changed: get_dynamic_global_properties Adding: get_nai_pool Changed: get_transaction_hex Changed: get_witness_schedule Changed: list_accounts Changed: list_comments Changed: list_escrows Changed: list_limit_orders Changed: list_owner_histories Changed: list_savings_withdrawals Changed: list_sbd_conversion_requests Adding: list_smt_token_emissions Adding: list_smt_tokens Changed: list_vesting_delegation_expirations Changed: list_vesting_delegations Changed: list_votes Changed: list_withdraw_vesting_routes Changed: list_witness_votes Changed: list_witnesses 29 methods added/updated in database_api Definitions for: follow_api, methods: 10 Definitions for: jsonrpc, methods: 2 Definitions for: market_history_api, methods: 7 Definitions for: network_broadcast_api, methods: 2 Definitions for: rc_api, methods: 3 Changed: find_rc_accounts Changed: get_resource_params Changed: get_resource_pool 3 methods added/updated in rc_api ``` This is how I can be ahead of any changes that might roll out. In this example, there seems to be a lot of changes, but most of them are just method signatures with keys in different orders. The most interesting results, to me, are things like `Adding: get_nai_pool`. If we focus only on that result, here's what we see newly added to `database_api.yml`: ```yaml - api_method: database_api.get_nai_pool purpose: parameter_json: "{}" expected_response_json: '{"nai_pool":[]}' ``` What I usually do at this point is search for the new method in the `steemd` repo. In this case, that leads to [database_api.cpp]( https://github.com/steemit/steem/blame/55a279769a4edebb2681e1c027ed8236e711402d/libraries/plugins/apis/database_api/database_api.cpp#L1456) and pull request [#3058](https://github.com/steemit/steem/pull/3058). I keep working backwards until I find the reason the method was added. Actually, this one is a little tricky. Looks like I need to investigate issue [#2886](https://github.com/steemit/steem/issues/2886). But I'll do that later. ### Test Curl The `rake test:curl[apis]` task is like a smoketest for the curl examples in the API Definitions. The rationale is that if we assume these curl examples are correct, then testing them periodically should show us problems in the API and/or node configuration. Conversely, it could also tell is if our documentation is out of date. Either way, running this rake task will give us a hint on how out of date documentation is. The tests also accept node options, so we can test against pre-release/testnets and get ahead of any changes that might happen before they go to production. ### Ops Dump As mentioned in the @steemitblog post from yesterday, we have `rake ops_dump`. When we run it, we get: ``` account_create account_create_with_delegation account_update account_witness_proxy account_witness_vote cancel_transfer_from_savings challenge_authority change_recovery_account claim_account claim_reward_balance comment comment_options convert create_claimed_account custom custom_binary custom_json decline_voting_rights delegate_vesting_shares delete_comment escrow_approve escrow_dispute escrow_release escrow_transfer feed_publish limit_order_cancel limit_order_create limit_order_create2 pow pow2 price prove_authority recover_account report_over_production request_account_recovery reset_account set_reset_account set_withdraw_vesting_route transfer transfer_from_savings transfer_to_savings transfer_to_vesting vote withdraw_vesting witness_set_properties witness_update ``` These are the broadcast ops that the devportal currently knows about, not including virtual ops. Broadcast ops are typically more important to app developers than virtual ops because they represent actions that applications can perform, whereas virtual ops are actions that the blockchain performs, typically delayed actions based on timestamps and other internal criteria. Once we know about a new op, we can add it to the `broadcast_ops.yml` file which Jekyll uses to generate: https://inertia186.github.io/devportal/apidefinitions/broadcast-ops ### Test Proof The `rake test:proof` task has Jekyll build the site, then run basically a lint check on the site. This allows us to find problems with the site. I also use [Site Sucker](https://ricks-apps.com/osx/sitesucker/) to further validate the results. ### Other Tasks The other tasks, like `rake production:deploy` and `rake production:rollback` are just used to build the site for use on production environments or switch back to the previous version. We need to do this externally instead of relying on Github Pages because there are various non-standard libraries required to build the site and Github doesn't like running non-standard libraries. We also have scrape tasks that will pull tutorials in from other repositories. #### Resources * To get a local copy of the devportal and try these tasks yourself: https://github.com/inertia186/devportal/blob/master/readme.md #### Previous Posts of this Series - [More Current Steem Documentation](https://steemit.com/utopian-io/@inertia/more-current-steem-documentation) (original fork announcement) - [Documenting Reputation and Database API Definitions](https://steemit.com/utopian-io/@inertia/documenting-reputation-and-database-api-definitions) - [Full Node Quickstart](https://steemit.com/utopian-io/@inertia/steem-documentation-full-node-quickstart) #### Github https://github.com/inertia186