The arguments for and against the various big blocks hard forks do get heated very fast. I tried to make my mind by reading blogs and watching interviews. After giving this considerable time and effort and doing my best to learn the arguments on the both sides I have made my mind. However, I realize that such an effort can take a lot of time, so I summarized the points that are most important in my opinion.

My conclusion is - while I do want lower fees and fast confirmation times, the price to achieve these goals by introducing big blocks is not justified. Big blocks is a fast solution that will probably work, but introduce new attack vectors on the Bitcoin network and more centralization. Moreover, it redistributes the influence over consensus from the full validating nodes to the miners which proved they have short term goals and will mine the chain that provides more profits at the current second.

Johnny vs. Roger Ver BTC vs. Bitcoin Unlimited

(Roger) Bigger blocks are tried and safe way to solve the issue of high fees and transactions stuck in mempool for hours/days. This is also the original roadmap as intended by Satoshi Nakamoto. The core devs (especially those from Blockstream), came much later and hijacked the original vision and changed it in a way that makes it harder or next to impossible to use Bitcoin as cash.
(Johnny) The fundamental information we learned about the system is entirely different right now relative to what where it was before Satoshi left. We learned about block attacks, mining pools, ASICS... The characteristics of the system are different.
(Johnny) There's infinite demand for free storage, no size of a block will be enough to satisfy the demand. Miners are incentivized to increase the block size. However, the full node maintainers are going to pay. This will create centralization in mining and full node maintaining.
(Johnny) It is possible to create "attack blocks" - blocks constructed in such a way that it will take a very long time to validate the transactions. With 1MB blocks - and attack block can take half a minute to validate. With 2MB blocks - about 15 minutes. With 8MB blocks - hours or more. SegWit partially addresses this issue, so 1-4MB blocks become relatively safe, but still, big blocks are unsafe.

Tone Vays vs. Roger Ver BTC vs. Bitcoin Unlimited

(Tone) There's no need for block increase right now, as SegWit + second layer protocols can solve the scaling issue entirely. If they fail though - it is possible to fork then with a lot less contention.
(Tone) Bitcoin Core has the best devs. It is a large decentralized team of developers. They are working together to create the best code for Bitcoin. They will either continue to work on Core or get pushed out and leave entirely. This is a very important reason to avoid hard forks.

Richard Hearts vs Roger Ver BTC vs. Bitcoin cash

(Richard) Having one implementation of the blockchain is better than having several. It is exponentially harder to avoid bugs in several implementations. In case of several implementations, every bug creates split of consensus. In such a case, it is required for a social consensus to step in and decide who is right. That creates situations where powerful entities can influence which way to go - like it happened with ETH vs. ETC. Governments can abuse the existence of such powerful entities (arbiters).
(Richard) Large blocks are the reason we have empty blocks. Empty blocks are created by miners mining on headers only while they still download the block. One more reason for a miner to create an empty block is to make sure that his block propagates faster through the network thus decreasing probability to become an orphan in case there's another full block propagating that was found more or less at the same time. In fact, these are DoS attacks. Miners are paid by the network to validate the transactions and put these into blocks - thus empty blocks are a kind of breach of the social contract between the users and miners.
(Roger) Empty blocks still help the network by making the chain longer and securing it with PoW. It may be being good because it provides the time to the network to catch up. The best way to achieve miners decentralization is by increasing the numbers of Bitcoin users and transactions. The more users there are - the more people will be mining thus creating decentralization.
(Roger) Bitcoin is losing market share to competing for crypto currencies due to high fees and long confirmation times. Provides an example where BitCards stopped accepting Bitcoin and now only use Dash and Ethereum. Roger argues that there is a significant amount of commerce happening on Bitcoin network and Bitcoin scaling issues harm this activity.
(Richard) Miners proved by switching the hash rate from Bitcoin to Bitcoin Cash they only chase the profitability and cannot be entrusted with changing Bitcoin consensus, i.e. hard forking.
(Richard) Core devs are the most professional dev team and with the deepest knowledge on how to scale Bitcoin without compromising on decentralization and security. They are self-motivated and are not paid by Bitcoin investors. Therefore, it is naive to assume they will automatically move to support another fork in case it becomes more popular.
(Roger) The rise in Bitcoin Cash value has nothing to do with Roger. Roger didn't sell any bitcoins/altcoins to pump BCH.
(Richard) The hard fork was done in an unprofessional way. The only wallet that allowed splitting was hosted on a centralized site. The wallet was signed by an unknown key. It forced a lot of people to download this software and use it insecurely. Moreover, it forced people to move old bitcoins from one address to other in order to split thus causing coins age loss and potential privacy loss.
(Roger) Bitcoin Cash was a surprise for him too. It is not his baby, and he is related but not the force behind it.