[Official] Findora Announcement Channel
Join us in ~24 hours for an X/Twitter Space on FRAgments: BRC-20 Inscriptions on Findora!
Without realizing it, Findora created a UTXO network that is perfect for inscriptions. Our technology arrived ahead of its time, but we are now moving quickly to deploy "Fragments." Fragments are inscriptions on Findora that are BRC-20 compatible.
We are issuing an upgrade to Findora custom asset codes that will make them compatible with the BRC-20 JSON data. We will also launch a web-based token inscriber and marketplace for users to mint and trade Fragments. We’ll soon be launching a new browser wallet to allow users to interact with these new tools easily.
Set a reminder and learn more by visiting the official announcement on X: https://x.com/Findora/status/1737898249640640837
EVM Staking Update - December 5th | 11:30 PM PST
This is likely the penultimate update in the EVM Staking saga 🎉.
Tomorrow, the new EVM Staking portal will be live, making it easier than ever to stake FRA and manage which validators you stake with.
Additionally, the engineers have rolled out an optional upgrade, v0.4.4, which allows validators to manually claim their commissions without waiting for delegator to claim their rewards. The final functions will be added to the toolbox in the coming days.
More details on EVM Staking, including a Postmortem with a message from Sam, will be released in the coming days as well. Through the process, Discreet Labs updated internal processes based on lessons learned, like the need for:
- Thorough internal and external QA testing
- Better internal communication with engineering
- Better external communication with the community
We want to sincerely thank the community for being understanding and patient. Switching network security from a UTXO-based ledger to the EVM ledger has not been done before because almost no other chains have a dual-layer architecture similar to Findora’s. Once again, we know it has been a trying process, and we are truly grateful for the community sticking by us in this time.
Stay tuned for more details on EVM Staking and a way to leave feedback on it.
Brief Update - November 27th | 9:15 PM PST
Summary:
Our top priority is fixing an issue affecting validator commission collection. The fix will be in the form of a mandatory update, with details to be announced once we gather more information from the Engineering team.
We're also in the process of collecting feedback from a small group of volunteer QA testers helping us iron out issues with the EVM staking portal. The time provided by those volunteer testers, mostly validators, is genuinely and greatly appreciated.
++++++++++++++++++++++++++
Top Priority: Creating a fix for validators to claim their rewards without issues. The timeline will be provided to the validator community as soon as possible.
EVM Portal Fixes: We will collect and prioritize the feedback from the QA testing feedback this week.
++++++++++++++++++++++++++
Please feel free to contact Daniel Pagan (@danielpagan) and Daniel Finley (@danielfindog) if you have any questions. Please remember that we will not DM you first for security reasons.
Brief Update - November 15th | 10:15 PM PST
Current Status:
The iOS and Android Mobile wallets have been updated. The latest mobile version, v0.4.4, seems to have fixed the bugs mentioned in the last update, and though there have been a few reports of the app crashing, the new mobile wallet seems to be running reliably for the community overall.
The actual APY earned by delegators is still low, but the engineering team hopes solve this issue by early next week. However, it could be solved as early as tomorrow, November 16th.
++++++++++++++++++++++++++++++++
Updates
APY Rewards — Engineers are reviewing the smart contracts to ensure that the APY rate returns to its proper amount according to Findora’s tokenomics. They hope to have a fix deployed by early next week, though it could be done as early as Friday.
Mobile Wallets — The new wallet version, v0.4.4, is running smoothly and seems to have fixed the bugs previously reported by the community. There are a few reports of the iOS version crashing which are being investigated to see if the issue is reproducible.
Full Nodes — The full nodes are now running reliably thanks to new redundancies, a monitoring system, and restructuring of the database.
++++++++++++++++++++++++++++++++
Note
Since most of the issues that arose due to the EVM Staking upgrade have been fixed, the team is turning more attention to QA testing the new staking platform and hopes to release it for third party testing soon.
As a result, we will begin issuing updates only once a week, but please feel free to get in touch with Daniel Pagan (@danielpagan) and Daniel Finley (@danielfindog) if you have any questions. Please remember that we will not DM you first for security reasons.
Brief Update - November 13th | 10:35 PM PST
Mobile wallets have been updated for iOS and Android. The latest mobile version, v0.4.4, includes several bug fixes listed below.
++++++++++++++++++
Mobile Wallet 0.4.4 Bug Fixes
Apple iOS:
— Resolved a display issue for the staked and total staked amounts when unstaking.
— Fixed the decimal place of the staking and unbonding amounts.
— Resolved an issue where users cannot click the Max button when unstaking.
— Resolved an issue where users, when unstaking, can only select the first validator listed.
Android:
— The calculation for the unbonding amount is now accurate.
— Resolved a display issue for the commission rate on validators.
++++++++++++++++++
If you're experiencing issues with your Findora wallet, please ensure you've updated and are running the latest release.
In the meantime, please contact Daniel Pagan (@danielpagan) and Daniel Finley (@danielfindog) if you have any questions. Remember that we will not DM you first for security reasons.
Brief Update - November 8th | 11:45 PM PST
Current Status:
The engineering team has implemented a fix that allows delegators to undelegate from offline validators. They also believe they will be able to deploy a fix that allows validators to manually claim their commission next week.
The team is continuing QA testing on the EVM Staking Portal and making adjustments on the front end. The engineering team believes that it will be ready for 3 party testing within a week.
Mobile wallets are still suffering display issues, but the the desktop wallet is stable, and we encourage users to use the latest version of the desktop wallet, v0.4.6.
++++++++++++++++++++++++++++++++
Updates
Delegator Rewards — Delegator rewards have returned to ~58% APY after being far lower at the end of the last week. While engineers work to normalize the APY, the elevated rewards will be a bonus to the community. They are also able to undelegate from offline validators.
Validator Rewards — Though validators are still earning rewards, they are not being automatically distributed to their wallet. The engineering team is working on a solution that will allow validators to manually claim rewards that will hopefully be available next week.
Mobile Wallets — tTe iOS and Android wallets are still suffering display and reliability issues. We are encouraging community members to use the Desktop wallet for the time being.
EVM Staking Portal — The team has completed a number of front end improvements and is continuing internal QA testing as they prepare the portal for 3rd party testing as well.
++++++++++++++++++++++++++++++++
In the meantime, please get in touch with Daniel Pagan (@danielpagan) and Daniel Finley (@danielfindog) if you have any questions. Please remember that we will not DM you first for security reasons.
Brief Update - November 7th | 1:53 PM PST
(Apologies for the delayed update; I had intended to send this last night.
- Daniel Finley)
Current Status:
The engineering team is working on internal QA testing for the new EVM Staking portal and has implemented a number of improvements. Those test will continue before being released to a 3rd parties to help conduct QA testing.
Engineers continued to make progress in creating a way for validators to manually claim their commission and fixed and issue that had reduced delegator rewards. They are investigating a way to normalize the APY but in the meantime, the high rate will serve to reward the community for their patience.
No fix for the mobile wallets is yet deployed, but the desktop wallet is accurate and stable. While the engineering team works to solve display issues on the mobile wallets, we encourage the community to switch to the desktop wallet for the time being.
++++++++++++++++++++++++++++++++
Updates
Validator Rewards — Though validators are still earning rewards, they are not being automatically distributed to their wallet. The engineering team is working on a solution that will allow validators to see the total rewards they are accruing and claim them.
Delegator Rewards — Delegator rewards have returned to ~58% APY after being far lower at the end of the last week. While engineers work to normalize the APY, the elevated rewards will be a bonus to the community.
Desktop Wallet — The latest version of the desktop wallet, v0.4.6., is stable, though a little slow and displays accurate information.
iOS Wallet - Engineers are still working on an issue where users sporadically encounter the error, "Exceeding the maximum single validation." We anticipate this issue being fixed by early next week.
Android Wallet - Some users are reporting that they are receiving inaccurate staking data. The engineering team is investigating to find the source and hopes to solve it with the stability fixes with the iOS wallet.
EVM Staking Portal — The team has completed a number of front end improvements and is now starting on internal QA testing and putting together procedures for external testing as well.
++++++++++++++++++++++++++++++++
In the meantime, please get in touch with Daniel Pagan (@danielpagan) and Daniel Finley (@danielfindog) if you have any questions. Please remember that we will not DM you first for security reasons.
Brief Update - November 3rd | 11:15 PM PST
Current Status:
The engineering team continues to work to improve the performance of the desktop wallet which is currently slow, but stable. They are also investigating display issues with the Android wallet and hope to deploy a fix for the stability issues with the iOS wallet by early next week. The team is continuing QA testing on the EVM Staking Portal.
++++++++++++++++++++++++++++++++
Updates
Validator Rewards — Though validators are still earning rewards, they are not being automatically distributed to their wallet. The engineering team is working on a solution that will allow validators to see the total rewards they are accruing and claim them.
Delegator Rewards — Community members noticed today that delegators seem to be earning lower rewards than they should. The team is investigating the reports and working to isolate a cause.
Desktop Wallet — A new version for desktop is available, v0.4.6. This version resolves an issue where users were unable to unstake from offline validators.
iOS Wallet - Engineers are still working on an issue where users sporadically encounter the error, "Exceeding the maximum single validation." We anticipate this issue being fixed by early next week.
Android Wallet - Some users are reporting that they are receiving inaccurate staking data. The engineering team is investigating to find the source and hopes to solve it with the stability fixes with the iOS wallet.
EVM Staking Portal — The team has completed a number of front end improvements and is now starting on internal QA testing and putting together procedures for external testing as well.
++++++++++++++++++++++++++++++++
In the meantime, please get in touch with Daniel Pagan (@danielpagan) and Daniel Finley (@ danielfindog) if you have any questions. Please remember that we will not DM you first for security reasons.
Brief Update - November 1st | 10:25 PM PST
Current Status:
Desktop wallets continue to show stability in displaying staking information, though data is still loading slowly. The team continues to work on new RPC endpoints that are optimized for EVM related traffic. These will help improve the performance of the wallet and make the EVM Staking portal reliable and stable for users when it goes live.
The team is also starting quality assurance testing on the EVM Staking platform, now that a solution to performance issues is imminent.
++++++++++++++++++++++++++++++++
Updates
Validator Rewards — The team has discovered the issue with validator commissions and is now working on a solution. Though validators are not able to claim or see their rewards, they are still earning them. Engineers are working on a two-part solution so that validators will 1.) soon be able to see the rewards they are accruing and 2.) be able to claim their past commissions.
Desktop Wallet — A new version for desktop is available, v0.4.6. This version resolves an issue where users were unable to unstake from offline validators.
iOS Wallet - Engineers are still working on an issue where users sporadically encounter the error, "Exceeding the maximum single validation." We anticipate this issue being fixed by early next week.
EVM Staking Portal — The team has completed a number of front end improvements and is now starting on internal QA testing and putting together procedures for external testing as well.
++++++++++++++++++++++++++++++++
In the meantime, please get in touch with Daniel Pagan (@danielpagan) and Daniel Finley (@ danielfindog) if you have any questions. Please remember that we will not DM you first for security reasons.
The next update will be shared Friday evening PST.
Brief Update - October 30 | 10:30 PM PST
Current Status:
Desktop wallets continue to show stability in displaying staking information. The team is working to resolve an issue with newly built RCP endpoints optimized to handle only EVM related traffic. Once resolved, the Findora wallet will be redirected to use the new endpoints, further improving the performance of the wallet.
The team is also looking into potential issues with validator commission and ensuring validators are receiving the proper amount of rewards. Updates for validators will be shared in the validator support channel on Telegram.
++++++++++++++++++++++++++++++++
Updates
Desktop Wallet — A new version for desktop is available, v0.4.6. This version resolves an issue where users were unable to unstake from offline validators.
iOS Wallet - Engineers are resolving an issue where users sporadically encounter the error, "Exceeding the maximum single validation." We anticipate this issue being fixed this week.
EVM Staking Portal — The team is working on a handful of front-end improvements on the staking portal. Once this is completed over the next few days, additional internal QA testing will be performed followed by external QA testing. We will keep everyone posted as progress is made.
++++++++++++++++++++++++++++++++
In the meantime, please get in touch with Daniel Pagan (@danielpagan) if you have any questions.
The next update will be shared by Wednesday evening PST.
Brief Update - October 25 | 11:50 PM PST
Current Status:
DevOps installed a new update to Findora full nodes. The new release, v0.4.3-4-release, brings stability improvements, which means a more reliable display of staking data on the Findora wallet. The time it takes to display staking data is longer than ideal, but users should see fewer instances of zero balances and timeouts.
The team will continue to monitor the full node performance while a more permanent solution is developed.
++++++++++++++++++++++++++++++++
Desktop Wallet — Staking data and balances should be displayed more reliably. Some timeouts (zero balance) may still be experienced, but this should occur significantly less frequently than before.
iOS Wallet — Some users are experiencing an issue where staking results in an error, "exceeding the maximum single validation." This was escalated to an engineer yesterday, and we're hoping to have an update for everyone by Monday night PST.
EVM Staking Portal — If the full nodes continue to show stability, testing, and additional improvements will resume on the EVM staking portal on Monday. Additional testing will involve volunteer validators not long after internal testing concludes.
++++++++++++++++++++++++++++++++
In the meantime, please get in touch with Daniel Pagan (@danielpagan) if you have any questions.
The next update will be shared by Monday evening PST.
Brief Update - October 25 | 11:50 PM PST
Current Status:
The balance and staking data display in the Findora wallet is sporadic due to backend issues with full nodes. Users may see balance and staking data in the wallet if the query reaches a working full node. The wallet will display zero values for other users if the query reaches a non-working full node.
Tokens are safe, though the symptoms may prevent staking features from being used.
Update:
The engineering team is testing and finding solutions to ensure wallet data is displayed more reliably and the backend platforms are more stable. We will likely not have much to share over the next couple of days while the team works on the issue.
In the meantime, please get in touch with Daniel Pagan (@danielpagan) if you have any questions.
The next update will be shared in 48 hours.
Brief Update - October 25 | 12:05 AM PST
Findora Desktop Wallet
Yesterday, Findora engineers released a new Findora desktop wallet, v0.4.5 which originally used Web3 endpoints. Though the Web3 endpoints enabled data to load quickly, using those endpoints caused a cosmetic issue where extraordinarily large unclaimed reward amounts were being displayed. To solve this issue, the DevOps team redirected traffic back to the Findora full nodes.
Using the full nodes allowed for accurate data, but the full nodes have become burdened, causing staking data to load slowly. Our DevOps team is reloading the node data now, which is expected to catch up in the next 72 hours or so.
Findora Mobile Wallet
Engineers have addressed an issue on the Findora mobile wallets that prevented the staking screen from loading. The latest mobile wallet version, v0.40.1, should solve this issue for all users, though we continue to monitor the mobile wallet's performance. Note that clearing the cache on the Android wallet can also help keep pages from freezing.
TLDR: The team has made progress in improving both the mobile and desktop wallets. Though data loads slowly on the desktop wallet, the data is accurate and engineers are hopeful a permanent fix is around the corner.
Brief Update - October 24 | 12:55 AM PST
Findora Wallet
A new update is available for the FIndora wallet, v0.4.5. This new version contains an update to the RPC URL and was originally set to use new, multiple Web3 endpoints when it was published, increasing redundancy for data retrieval. However, a cosmetic error was quickly noticed which drove DevOps to redirect the RPC traffic to Findora full nodes instead.
Engineers are resolving the cosmetic issue on the Web3 endpoints which should be done relatively quickly. Once that's done, DevOps will redirect wallet RPC traffic back to the Web3 endpoints via load balancer settings.
This new wallet update is available for Windows and will soon be available for other platforms as well. We'll share more information on their progress in our next update to be shared within 24 hours.
TLDR: New wallet version 0.4.5 is available for Windows, coming soon to other platforms. RPC URL for the new wallet was updated for more redundancy but a cosmetic issue was found and soon to be resolved.
Brief Update - October 20 | 10:20 PM PST
Graph Node & Desktop Wallet
The Graph node has been more stable these past 48 hours compared to the performance impact we've observed these past couple of weeks.
This means the desktop wallet data display has been more reliable. However, the wallet may sporadically fail to display staking data until a permanent fix is created. While not the most ideal, the sporadic issues are temporary, and the improvements are a step towards stability and reliability.
In the meantime, the permanent fix - restructuring of the Graph node database - is expected to take approximately two weeks.
EVM Staking Dashboard
Assuming we continue seeing improvements on the Graph node, we plan to finish the EVM staking dashboard and resume QA testing next week.
iOS Mobile Wallet
An engineer is continuing to work on the iOS wallet. Currently, the staking screen becomes stuck on "Loading." We're hoping to have this resolved by early to mid next week.
——————————————
Note: Daily updates will resume on Monday.
pinned «Update | October 19, 9:00PM Summary: The team continues to make changes to make the Graph node more stable. As shared in previous updates, we have doubled the size of the VM and upgraded the IO speed of the database disk. Though the desktop wallet has shown…»
Update | October 19, 9:00PM
Summary: The team continues to make changes to make the Graph node more stable. As shared in previous updates, we have doubled the size of the VM and upgraded the IO speed of the database disk.
Though the desktop wallet has shown significant improvement today, we noticed that staking data sporadically failed to display. However, its performance today was far more reliable than previous days.
We’re hoping the updates below add to the Graph node’s stability as we implement a permanent fix.
Updates
Graph Node 🟢
We are also making the EVM Staking portal more efficient. Previously, when the portal loaded, several queries were simultaneously sent to fetch validator data off the Graph node. We have reduced that to a single query instead, which should reduce unnecessary load and improve Graph node performance.
We believe these improvements add to the stability of the Graph node and serve as a workaround. As a long-term solution, engineers are strategizing to restructure the database. This work will optimize how specific tables are structured, further enhance the Graph node’s performance, and provide a more permanent solution.
iOS Wallet 🟡
The staking screen on the iOS wallet app is still experiencing an issue where it gets stuck on the “Loading.” We have assigned an engineer to resolve the issue, and we are hopeful it will be fixed soon.
TLDR: With improvements being made to the Graph node and iOS wallet app, EVM Staking is getting much closer to being ready for its public launch.
October 18 - 10:25 PM
Update: The Graph node has been far more stable since yesterday’s update. However, despite the improvements, there was a brief period today when the Graph node became overburdened.
To address this, the engineering team has doubled the size of the VM today. This temporary solution should bring additional performance improvements of the Graph node as we work to implement a more permanent fix.
Meanwhile, we are aware of an issue with the iOS version of the Findora wallet app where the staking screen becomes stuck on "Loading." We have found that if other pages on the app freeze, reinstalling the app and importing your private keys tends to fix the issue. An engineer is tasked with running diagnostics to find the underlying problem of the stuck staking screen.
We are looking forward to finally focusing on testing and launching the new EVM Staking platform. Validators on Findora, if they’re willing, will be asked for their time to participate in additional testing of the new portal before it goes live. We know it has been a trying time, and we appreciate everyone’s patience and understanding.
TLDR: We have doubled the VM size to stabilize the Graph node as we look for a permanent fix (workaround); An engineer has been tasked with fixing the Findora wallet iOS app (work in progress); EVM Staking portal is coming soon.
Update | October 17 - 10:35 PM
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟢 — Tentatively
Work in Progress
Correct Data Now Displaying
Update: Yesterday, we announced that a certain query was causing our Graph node to lock up because it required an incredible amount of resources from the server. Since last night, engineers have implemented a few different temporary workarounds which have resulted in significantly improved stability in the Findora wallet.
First, they have added redundancy through a backup node to reduce the recurrence of the issue. They have also upgraded the IO speed of a database disk and improved the VM, which have contributed to the stability of the Graph node seen today. The team will monitor the effectiveness of these temporary solutions as it continues to develop a permanent fix.
The result of all these efforts is that today, users have seen far more reliable data being displayed in their Findora wallets.
Now that the Graph node is looking more stable, the engineering team is starting to shift attention to the EVM Staking portal, and will soon be conducting the final checks needed before its public launch. More details to be released soon.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Work around now implemented for the Graph node which is showing more stability. Permanent solution in the works (Work in Progress).
Update | October 16 - 10:55 PM
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟡
Work in Progress
Correct Data Sporadically Displays
Update: We found a query that’s causing our Graph node to lock up. This query requires an incredible amount of resources from the server. Engineers are working on reducing the resource requirements of the query and evaluating multiple solutions, choosing the one that doesn’t just resolve the issue today but also one that is most optimized and best for the long-term status of the service.
Optimization of queries and splitting certain database tables are two options engineers are evaluating.
In the meantime, DevOps is working on additional redundancy methods to reduce the recurrence of this issue so users are no longer impacted while a permanent solution is created. This is where troubleshooting efforts are focused right now.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Graph node continues to exhibit issues leading to sporadic incorrect wallet displays. DevOps is working on a more stable workaround (Work in Progress).
Update [October 15 - 11:02 PM]
Graph Node
Symptom: Staking Values Missing in Wallet (Cosmetic)
Current Status: Workaround Being Applied
Update: DevOps built a secondary Graph node to serve in a hot-standby role, meaning the backup Graph server can serve requests should the primary fail as we continue to fix this particular problem. This added redundancy should help reduce or avoid impact on your wallet staking information.
Wallet staking data should display properly in a few hours once the new Graph node catches up to the current block height.
Full Node (DB Lock)
Symptom: No User Impact
Current Status: Workaround Applied
Update: We are continuing to diagnose what has caused the database to lock on the full nodes. We have a current workaround that proactively restores service when this happens, but we will apply a more permanent solution once the root cause has been established.
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: Resolved
Update: All 113 unbonding transactions have been processed, and users whose unbonded tokens were missing should now show in the Findora wallet.
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: Resolved
Update: All Rialto Bridge transactions are now reflecting properly in user balances.
Update [October 13 - 11:30 PM PST]
Two fixes are top priority for engineers.
1. Unbonded Tokens Not Displayed in Wallet
2. Staked FRA Prior to Upgrade Not Displayed in Wallet
These issues are summarized in item 10 on the tracker. See previous pins for the latest tracker update.
After making some progress with the Graph node and full nodes, the Engineering team has redirected their troubleshooting and improvement focus to staking contracts. They've identified an issue that, when resolved, will have the Findora wallet display proper staking information.
The Engineering team created a fix which is under review by one of our smart contract developers. They are working together on reviewing the code which will be applied as soon as the review completes.
_________________________________
While this is a cosmetic issue, and all funds are safe, the inconvenience and concern expressed from the community is heard and understood. The team will work hard to improve our processes and share a retrospective once all priority items are addressed.
We are coming close to having all wallets fully functional and properly reflecting your balance. Thank you and we will share another update in 24 hours at the latest, though updates might be shared sooner.
_________________________________
TLDR: Engineers redirected their focus to staking contracts, found an issue, and have created a fix. The fix is under review by another smart contract engineer. If no issues are found in the code, and with the latest progress made on the Graph node and full nodes, the fix will be applied and all wallets will display correct balances.
Update [October 12 - 10:35 PM PST]
Graph Node
The Graph node continues to show stability and correct data. This stability is allowing the EVM staking portal to reliably show valid stake information. While additional QA testing will be needed on the EVM staking portal, this good news brings the portal closer to public readiness.
Full Node
We are continuing to test newer versions for the full nodes, ironing out bugs and improving stability to ensure they do not fall behind. In the meantime, DevOps implemented a workaround that ensures all full nodes are operational and responsive, proactively preventing any impact the full nodes would have on the community. This workaround also ensures that the engineering team can approach their work cautiously while mitigating user impact.
_______________________
Next Steps ⤵️
_______________________
Wallet Unbonded Tokens
Now some good news...
With the Graph node showing stability and the full node workaround in place, unbonded tokens should be reflected properly in user wallets within the next 24 hours. No additional wallet upgrades will be needed. If you’re on a desktop, simply ensure you run the latest release v0.4.4.
Note: Staking FRA and claiming rewards should currently be functional in the Findora wallet. This fix is specifically for unbonded tokens.
Missing Bridged Assets (Rialto)
With the Graph node showing stability and the full node workaround in place, any users who bridged tokens over Rialto and could not retrieve them will find those tokens returned to their wallet in the next 24 hours.
________________________
Below you'll find the latest update to our tracker. We anticipate for the last item in the list regarding unbonded tokens to be resolved within the next 24 hours. I've also added notes for additional context on each item.
_________________________
TLDR: Unbonded tokens should appear in your wallet within the next 24 hours. Staking and claiming in the Findora wallet should currently be functional. Graph node is stable. Full node has a workaround to mitigate impact while engineers continue to work on it. Any missing tokens bridged over Rialto will be shown in the originator's wallet in the next 24 hours. EVM staking portal is closer to public readiness but needs QA after our retrospective is published.
Update [October 11 - 10:45 PM PST]
Full Nodes
The DevOps team is testing a newer version that was provided since last night’s update. This version is in testing, and so far, the release appears to be stable.
If the team continues seeing stability, this version will be installed on all full nodes. Validators will not need to update. This is only applicable to full nodes. In the next 24 hours, we’re hoping to deliver news that all full nodes have been or are in the process of being upgraded.
Graph Node
We are seeing significant improvements on the Graph node since our previous update. The engineering team is monitoring the Graph node’s performance, and no issues have been observed in the past several hours. Previous observations showed failure events occurring in short succession of each other, so this is a great sign of progress.
If the Graph node continues showing stability, we can expect the in-wallet staking issues to be resolved within the next 24-48 hours. Like our update on the full nodes, we hope the next update will reflect a working and predictable balance and staking values in the wallet.
Note on Rialto
These issues have also impacted Rialto bridge transfers. Fixes for the Graph and full node should bring Rialto back into full service.
Next Steps
Please continue to monitor the pinned messages for updates. Once these issues are behind us — which should be very soon based on tonight's observations — the engineering team will begin drafting an RCA, which we will share with the community. We will also begin to draft a document detailing changes to our upgrade rollout process, including improved QA testing, internal escalation steps, and more.
We will have an updated Issue Tracker for the community in tomorrow's community update.
Update: [October 10 - 10:00 PM]
Full Node
Last night, we received a newer build for Findora full nodes. However, issues were detected and resolved during testing. DevOps now has a newer build for full node in testing, and all appears to be going well. The team aims to upgrade the full nodes in the next 24 hours.
Note: This full node update will not resolve the incorrect staking balance. However, it will bring added stability to the network and its applications, such as the Findora wallet.
Graph Node
The engineering team has eliminated several potential root causes, allowing them to narrow their focus. They are continuing their debugging efforts. Once the root cause is identified, we can expect a solution in a speedy manner.
The team is working on these issues in real time, and hopes to provide more progress updates tomorrow.
Update: [Oct 9th - 10:00 PM PST]
Full Node
The DB lock issue on the full node is suspected to be resolved. However, testing is needed before installing this latest release on a production server. DevOps has the new version, which will go into testing tonight. This fix will improve UTXO functions, such as faster UTXO transactions.
If the new version tests well, the code will be installed into production in the next 24 to 48 hours.
Graph Node
The Graph node database shows an exceptionally high number of connections. This is leading to the Findora wallet displaying incorrect staking information. Engineering is reviewing more debugs off the Graph node to determine what is causing this behavior.
Bridged Tokens
Any tokens locked on the Rialto bridge will be released using a script being prepared by engineering in the next 24 to 48 hours. Tokens will be distributed back to the originating address.
Update: [Oct 7th - 10:40 PM PST]
The Engineering team continues working on resolving issues with the Graph node and full node.
With the Graph node, queries receive no response shortly after the server loads and indexes data. This is likely why testing the v0.4.4 wallet was successful when the previous update was written, but began to fail shortly after.
The full node is showing the local DB receiving lock requests from different processes, causing high system resource utilization and degrading the server's response time.
Fixing the issue for the Graph node and full node will very likely resolve most or all issues with the wallet. I have confirmed again the engineering team is giving full attention to these fixes, treating this with urgent priority.
We'll circle back to everyone with more information tomorrow afternoon (PST) and allow the Engineering time time to focus on identifying root causes and implementing fixes.
Latest Update [Oct. 6th | 8:50 PM PST]
1— FINDORA WALLET:
Engineering has resolved the issue where balances were not being displayed. This fix is contained in the new v0.4.4 release available on https://findora.org/#wallet. However, a new cosmetic/display issue has been identified where recently unbonded FRA is not reflected in the total balance. Engineering is working on a solution and prioritizing this as an urgent fix.
2— EVM STAKING PORTAL:
The DevOps team made changes which significantly improved response time on the new EVM staking portal. The team is working on a Graph node rebuild. This rebuild will improve response time for other components of the new staking portal which continue to experience a sluggish response time.
3— VALIDATORS:
The engineering team is still working on providing node runners with a script for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line. DevOps identified some discrepancies on validator profile pages, such as total staked vs self staked. These discrepancies will be resolved with the Graph node rebuild.
We have marked two additional issues as RESOLVED since yesterday’s update and plan keep the community updated over the weekend.