2024-04-08 An Update From the Implementer Group

Original post

This thread was brought to the implementers’ attention yesterday. After gathering more information and evaluating the situation and impact, implementers from Lotus, Forest and Venus, and some of the core devs think it is worth considering postponing the nv22 Dragon upgrade by 2 weeks. Given this delay, the new upgrade is scheduled to be on 2024-04-24 -14:00:00Z.

It is unusual to change the upgrade timeline at this last minute, and we don’t want to set a precedent here, however, we want to be sensitive to the fact that there is an important community of network power that may face challenges to update their nodes in time. The new upgrade epoch has taken Lotus’ team’s prescheduled traveling/OOO and Forest core dev’s OOO into consideration.

While the Dragon upgrade brings great features and enhancements to the network that implementers have worked hard on and users will benefit from greatly, this upgrade is not urgently time sensitive. Therefore, a short delay to increase the chances for the impacted users (token holders, SP operators, and the data clients they serve) to find the right resolution is worth considering as a community. We hope the community can use this 2 week delay to seek the best resolution and update their nodes ASAP.

The original upgrade epoch is coming soon, so all three implementations will aim to put out a patch with the new April 24th upgrade epoch ASAP for those who have already upgraded their nodes for the April 11th upgrade to avoid a forked version of the network and disrupted services. Please closely monitor the implementation release channels for latest updates.

P.S.: Note from @steven004
P.P.S: Lotus release v1.26.2 for this update has been published. Our top priority is to support the node operators in updating their nodes accordingly to avoid undesired system downtime. If you have any questions, feel free to reach out in #fil-lotus-help.

Update on Nov 2nd, 2023

Calibration

A bug in the move partition actor implementation was discovered during the testing, more details can be found here. Fortunately, we caught it on the testnet before it's launched on mainnet 🎊 And unfortunately, to fix this bug on calibration needs an upgrade as it is a consensus breaking change.
We have released actor v12.0.0-rc2 with the fix, and will schedule a WatermelonFix upgrade at epoch 1070494, 2023-11-08T11:00:00Z to rollout the patch for calibration.

Calibration participants should checkout implementation announcements and make sure your nodes are updated to the release respectively.

Mainnet

The mainnet upgrade is scheduled at epoch 3431940 , 2023-11-29T13:30:00Z.

Nov 16th, 2023 ❗ Important notice w.r.t nv21 scope and upgrade timeline!

Hey everyone!
👋 First off, a big THANK YOU to everyone in the community who has been involved in calibration network testing of nv21, and for your help in restoring it to a healthy state after last week's testnet chain halt.

😿 Unfortunately, FIP-0070 - Allow SPs to move partitions between deadlines has proven to be quite challenging to implement due to the complexity of the code, and additional testing uncovered an additional bug when lotus team was checking invariants in the Calibration-network. Given that the proposed Mainnet network upgrade was approaching fast, and the path for fixing this bug was unclear, the governance team started a nv21 decision matrix for FIP0070 that was brought forward to the core-devs.

FIP-0070 is descoped from nv21 upgrade

📣 Based on these discussions, the core developers have decided to descope FIP-0070 from the nv21 upgrade. This decision was made to ensure the security and stability of the network upgrade and ship other new features and protocol enhancements to network users before the year ends. FIP authors & implementers will do more iterations on the deisgn and implementation, and ship it in future upgrades.

Mainnet upgrade timeline

Mainnet upgrade date has been adjusted due to the rescope of nv21 & retesting on test network, as well as giving the community enough time to upgrade their infrastructure.
The new date and epoch for the Mainnet nv21 upgrade has therefore been set to epoch 3469380 - 2023-12-12T13:30:00Z.

Let's unpack a bit, what happened on Calibration testnet?

🐛 What is the bug? In simple terms, the bug is related to how miner actor track the expiration of sectors. We found that if a sector was moved from one deadline to another, its expiration date was not being updated correctly in all the necessary places. This could potentially cause a sector's expiration to be missed if the sector experienced a fault and then recovered. For a full technical breakdown of the problem, you can read through this thread.

❓ What does this mean for the calibration network? After careful discussion among implementers, we've decided to perform another miner CodeCID migration to remove FIP0070 from v12 actor code, so that the calibration testnet aligns the network protocol and state with what it will be when the mainnet upgrades to nv21.

❗ Please note that this migration is scheduled for epoch 1108174, which falls on 2023-11-21T13:00:00Z. Please check each of the implementers channel for which version you need to be on for this migration.

2023-11-22

Calibration upgrade was successful! Please follow your implementation announcement channels for the final releases!

2023-04-18 Expedited nv19 upgrade timeline

In light of the recent degraded chain quality on the mainnet an expedited nv19 upgrade has been proposed to roll out the market cron mitigation FIP0060 that will improve block validation times, and hopefully further improve the degraded chain quality currently.

We need all community's support to ensure an expedited but smooth network upgrade, we want to inform you of some key changes and important dates:

• nv19 upgrade on mainnet is moved up from May 11th to April 27th.
• Core devs have decided to postpone the activation epoch fixes and FIP0052 - Extend sector/deal max duration to 3.5 year. to a later network upgrade.
  • By descoping these changes we can greatly derisk the network upgrade itself by removing a heavy migration that could cause instability for storage providers and node operators during the network upgrade.
• Increase the rollover period for FIP0061 from 1 week to 3 weeks on mainnet. The rollover period is the duration between nv19 and nv20 which both old proofs (v1) and the new proofs (v1_1) proofs will be accepted by the network.

What does this mean for the calibration network?
Because of the expedited rollout to the mainnet we will have a shorter upgrade window between nv19 and nv20 on the calibration network. We will also have a shorter testing window from the calibration network upgrades to the mainnet network upgrades. Some key dates:

• 2023-04-18 feature code freeze in all reference implementation repositories.
• 2023-04-20T14:00:00Z at epoch 489094: nv19 Calibration upgrade
• 2023-04-21T16:00:00Z at epoch 492214: nv20 Calibration upgrade

All node operators and storage providers currently running on the calibration network please update your nodes to Lotus v1.22.0-rc4 before 2023-04-20T14:00:00Z.

We'd like to ask community members to actively test in calibration net and report any potential issue.

What does this mean for the mainnet network?

• The nv19-upgrade on mainnet from May 11th to April 27th! Tentatively 2023-04-27T13:00:00Z at epoch 2809800.
• The rollover period between nv19 and nv20 which both old proofs (v1) and the new proofs (v1_1) proofs will be accepted has been extended to 3 weeks. The nv20 “ghost upgrade” where the old proofs (v1) will now longer will be accepted by the network will happen on May 18th.

Please refer to your implementation announcement channel for release timelines.

Thank you for your understanding and cooperation. 💙

Confirmed Upgrade Schedule

Calibration

nv19 - 2023-04-20T14:00:00Z at epoch 489094
nv20 - 2023-04-21T16:00:00Z at epoch 492214

Mainnet

nv19 - 2023-04-27T13:00:00Z at epoch 2809800
nv20 - 2023-05-18T13:00:00Z at epoch 2870280

✅ Calibration upgrade nv19 is successful! nv20 coming in 22 hrs.

2023-02-06 update

📣 Due to the audit + development findings + FIP governance findings/process, the calibration upgrade will be postponed - we will provide an updated date later this week.

2023-02-08 Updated nv18 Hygge upgrade timeline is here!

Thank you all for your patience! After more 👀 and 👩🏻‍💻, here is our targeted (and hopefully final 🤞 ) nv18 Hygge Upgrade - FEVM launch timeline:

• Calibration Upgrade: 2023-02-21
• Mainnet Upgrade: 2023-03-14 (Yes, the Filecoin Mainnet Etherum chain id is 314 😉 )

Please see the Implementation section of the main post for implementation release timelines!

2023-02-14 Calibration nv18 Hygge Upgrade Update

calibration-net nv18 Hygge upgrade at epoch 322354, 2023-02-21T16:30:00Z. The bundle the network will be using is v10.0.0 actors, manifest CID bafy2bzaced25ta3j6ygs34roprilbtb3f6mxifyfnm7z7ndquaruxzdq3y7lo.
For lotus release info, please see here filecoin-project/lotus#10219

2023-02-16 Mainnet nv18 Hygge Upgrade Update

Mainnet nv18 Hygge upgrade is currently aimed at epoch 2683348, 2023-03-14 03:14 PM UTC. We will confirm the final built-in actor bundle CIDs in a follow-up notice later. Be sure to subscribe Filecoin status page for future reminders or any potential changes!

Bug 🕵️‍♂️ !! Calibration testnet will be reset!

we have discovered two migration bugs for nv17 upgrade during calibration testing , see more details here

🐛 Bugs in go state type migration:
1️⃣ we didnt do - qoute in fip Add the pending allocation mapping to market state, and claim ID field to deal states (all 0) , thus all deal states for acctive deals pre migration has a missing deals.
fix: filecoin-project/go-state-types#108
2️⃣ upon migration, we shall add datacap balance for all verified client that have pending deals in market actor
fix: filecoin-project/go-state-types#110

🤔 tl’dr impact if we didnt catch it during testing and the mainnet is upgraded with shark

• market cron will be borked due to have trouble to get deal state
• miner cron is borked with process_early_terminations for miner actors that have deals due to trouble getting deal state.
• miner actor that has active deals and tries to terminate sector/deal early will be removed from power table and lost all power

👍 Again fortunately we’ve caught it before it the main net upgrade!
However, the impact described above is effective on calibration, which was upgrade to nv17.
crons are having issues mentioned above and the calib is in a bad state that no longer simulates mainnet behaviour.
We believe the best thing for the network to do, is to RESET the calibration, accumulate states for ~5 days, and perform the upgrade again for 🦈 testing.

❗ That being said, the CalibrationNet is brought down on Oct 24th, 2:00 AM UTC and we will share more details about its reborn later!

📓 For Mainnet, due to most core dev teams will have reduced capacity until Oct 31st due to PLN labweek, we will postpone the Mainnet Shark Upgrade from Nov 9th to later in Nov, for more correctness testing.

Calibration-net has Reborn 🦋

Calibration net was reset successfully on Nov 1st, he network params can be found here (bootstrap, genesis car).

Calibration nv17 Shark 🦈 upgrade

The calibration Shark upgrade is scheduled at @epoch 16800, around Nov 7th, 14:10 UTC. We welcome all community members to update your nodes accordingly (lotus v1.18.0-rc5) and join the testnet for testing!

Mainnet Shark Upgrade

The upgrade epoch is scheduled to be at epoch 2383680, on 2022-11-30T14:00:00Z. All node operators shall update to the implementations accordingly before then to avoid lose sync with the network!

🌟 2022-06-14 Calibrationnet nv16 Skyr Upgrade

Calibration-net upgrade is set at epoch 1044660, 2022-06-16T17:30:00Z. The upgrade will use proof v11.1.0 & built-in actor v8.0.0-rc.1, bundle manifest CID bafy2bzacedrdn6z3z7xz7lx4wll3tlgktirhllzqxb766dxpaqp3ukxsjfsba.

As you may already know, this upgrade introduces non-programmable FVM M1 to the network and switch the network from using go spec-actor to rust built-in actor with new gas model - ALL BRAND NEW ⭐ 🔥 !

Given the number of changes being introduced for protocol + impl code base, we wanna encourage ALL network participants to join the implementation teams to test the new codes and catch potential consensus critical bugs or necessary enhancements in builtin actor, fvm, and client implementation. The more joint testing effort on calibration we get over the next 2-3 weeks as a community - the more confidence & guarantee we will get for having a smooth and safe upgrade later in mainnet as a network!

Note that - implementation teams should be responsive to the bugs being reported during the testing period. Based on the testing feedback, the mainnet upgrade will be scheduled within 2-3 weeks from the calibnet upgrade.

For lotus users, please follow this discussion for more detail, or you can join #fil-net-calibration-discuss in FC slack for questions!

🔥 2022-06-15 Mainnet nv16 Skyr Upgrade Announcement

The thing we all have been waiting for, FVM, it's finally coming! Currently the Mainnet nv16 upgrade is scheduled @epoch 1960320, 2022-07-06T14:00:00Z (10am EST).

Though this is a big change for the network, the Core Devs are not expecting any network disruption with this upgrade. Nevertheless, we'd like to proactively schedule a longer than usual maintenance window, and kindly ask the Filecoin community to join us monitoring the network post-upgrade, and stay reactive to any circumstances that may occur.

Please follow this thread and implementation announcements to get timely updates, and be ready for the upgrade!

OhSnap Notice to Filecoin Community

As many of you have been following along the core dev discussion around the upgrade scope and timeline in the past few weeks, you may have noticed that there is a last-minute change w.r.t to the testnet plan and I'd like to point out a couple things for our community.

Snap Snap Snap

Snap Deal is finally coming!! Many teams have been putting in a lot of hard work in the past 5 months into researching and designing how to incorporate datainto committed capacity sectors by the Filecoin storage providers with fewer steps, fewer resources and less time. From the initial proposal (a 3-message protocol), we have arrived at the final 1-message protocol that allows SPs to put valuable data right into existing CC sectors, by just doing one computation process of ProveReplicaUpdate. This also allows SPs to provide storage service that takes less than 1/5th of current data onboarding time - hopefully clients will appreciate and enjoy this as well 😉

New Proof Params

New circuits and proofs have been designed to enable Snap Deals, if you are interested in the specifications, you can find them there. What this means to the Filecoin network is that - we are introducing a set of new proof parameters for the very first time since mainnet launch! A trusted setup ceremony is run with ~20 participants from various teams from Dec 5th- Jan 26th . To learn more about what is trusted setup ceremony, please see this blog post from pre-mainnet and you can follow along this ceremony here or in #fil-trustedsetup in filecoin slack.

Butterfly-SnapNet

As mentioned above, this will be the first time that we introduce new proof params to the Filecoin Network, we would love to make sure that all related network participants are fully prepared and equipped to get the final proof params for the mainnet upgrade, by testing the flow on the most mainnet like network - calibration net. However, the final proof parameters won't available until the trusted setup is done, and without delaying the timeline for Snap Deals to be enabled in the Filecoin Mainnet, the lotus team has decided to work with the infra team, to reset the butterfly test network and turn it transform it to a SnapNet.

The goal of SnapNet is to provide a network for the implementation teams to test out the new code, for SPs to easily join the network, try out the new features and plan out the new potential service for the clients, and application/tool developers to update and test their products to adopt the new Snap Deal flow.

Note that the Butterfly-SnapNet will be running a set of temporary proof params and is a one-time-use network that likely will be reset after the the testing is done. We’d encourage the community to join this network, snap some deals and let the implementation teams know if you found any bug or have any suggestions for improvements!

Once the final proof params are available, we will be upgrading the calibration network and have it running for ~three weeks before the mainnet upgrade.

2022-02-08 Updates

1. Trusted setup was completed, checkout this blog post for more details!
2. Calibnet Upgrade is set to be on 2022-02-10T19:23:00Z @ epoch 682006!

Mainnet Network v15 OhSnap Upgrade

The upgrade epoch for network v14 OhSnap will be on epoch ** 1594680**, 2022-03-01T15:00:00Z, mark this date on your calendar and get ready for Snap!

Notice!

A bug was discovered in Rust actors that causes ProveReplicaUpdate messages to fail when sectors with different deadlines.

This is a small issue that will be fixed in the v16 network upgrade. Although it does not pose a security risk for the network overall, network clients should be aware.

For more information: filecoin-project/specs-actors#1577

maybe "the page previously at network.filecoin" (since that link now redirects)?

@jennijuju flagging that we are also considering FIP-0026: Fault Sector Extension

@jennijuju flagging that we are also considering FIP-0026: Fault Sector Extension

@kaitlin-beegle Do you mean #190, fip-0025 potentially? I will add it to the Highlight section as soon as the Draft FIP is officially merged in the FIP repo. Note: it is proactively implemented in lotus v1.12.0-rc1 as mentioned above.

Mainnet Network v14 Chocolate Upgrade

The upgrade epoch for network v14 Chocolate will be on epoch 1231620, 2021-10-26T13:30:00Z, mark this date on your calendar and get ready for 🍫 !

@jennijuju! yes, you're correct- the FIP number changed.

For clarity, the following FIPs have been accepted and are ready for implementation in Network v14:

FIP-0020: Add return value to WithdrawBalance
FIP-0021: Correct quality calculation on expiration
FIP-0022: Bad deals don't fail PublishStorageDeals
FIP-0023: Break ties between tipsets of equal weight
FIP-0024: BatchBalancer & BatchDiscount Post-HyperDrive Adjustment
FIP-0026: Extend sector fault cutoff period from 2 weeks to 6 weeks

🍫 is live!!

Calibration-net Reborn 🦋

Thanks to the Filecoin community, new features that will be introduced by network v13 and actor v5 are being tested excessively in the past few days. We have heard many positive feedbacks from miners, especially on how they will leverage the new PreCommitSectorBatch and ProveCommitAggregate to onboard more storage on Filecoin with less gas consumption🚀.

During the process, miners has provided valuable feedback on related configuration docs and reported unexpected behaviors. One of the reports has led devs from lotus and CryptoComputeLab to identify a bug in filecoin-ffi that would be problematic for huge messages getting processed in the chain from both gas and message size limit perspective. The issue has been identified and fixed here.

Unfortunately, to adapt this fix, it will be consensus-critical for calibration-net, which has upgraded to network v13 already. That being said, to properly run full testing again in the testnet, the dev teams have decided to reset the calibration net with the right proofs. This also means we have to re-examine HyperDrive params for FIP0008 and FIP0013 that depend on the size of proof aggregations, and then integrate the updated actor(if needed) and proof into implementations. You can track the work being done regarding this here.

Base on the test result, the complexity of the action needed may differ, at the moment, we are expecting the calib-net reset to happen on Jun 18th or Jun 21st, we will send an update here once we have a more firmed date. We are also enabling controllable VRK in calib-net upon the reset so that Filecoin plus features and verified deals can be tested on calib-net as well.

Once again, we highly appreciate everyone that has been testing on calib-net to ensure the upgrade goes smoothly on mainnet💙, and we will need your continuous support to keep testing on the reborn calib-net with the new implementation releases! As of today, we think this may delay the HyperDrive upgrade epoch by a week to early July. If you have any questions, leave a comment here or join and ask in #fil-net-calibration-discuss on Filecoin Slack!

Following up on the above thread, the calibration net will be retired within the next 3 hours. We will update you as soon as it's ready to start up again!

Calib-net has been reset at 01:00 am UTC, June 19, 2021. The upgrade epoch for network v13 HyperDrive on the post-reset calib-net is 420.

Mainnet Upgrade Epoch

Core devs have decided that the upgrade epoch for network v13 HyperDrive will be on epoch 892800, 2021-06-30T22:00:00Z, mark this date on your calendar and get ready for 🚀!

I'm super excited to share more details about the upcoming network 13 HyperDrive upgrade with all of you!!!! 💙 It will unlock a 10-25x* increase in storage onboarding rate due to innovations in Filecoin Proofs system! (*depending on how storage providers take advantage of the new aggregation limits), read more at https://filecoin.io/blog/posts/filecoin-v13-hyperdrive-network-upgrade-unlocks-10-25x-increase-in-storage-onboarding/

FIP-0015 has been accepted for this upgrade.

📣More testing is being done by core devs from different implementations to ensure all these new features land safely in the network. We are aiming to upgrade the calibration net in the next few days for the community to test and try out the new feature for ~2 weeks. That being said, the mainnet upgrade epoch is now postponed to the end of June, we will announce the exact upgrade epoch in this thread once it's decided by core devs!

Calibration net upgrade epoch has been added - @ epoch 321519 on 2021-06-11T14:30:00Z! Let the test begin!! 😻Things to try:

• PreCommitSectorBatch
• ProveCommitAggregate
• Top up a clients datacap multiple times!

Upgrade Epoch is Rescheduled to 712320

📣Based on community feedback, core devs have agreed to postpone the mainnet upgrade epoch to 712320 around 6 am UTC April 29th, which is 8 hours later than the previously planned timeline! Please see above for each implementation's release timeline.

Network upgrade reminder

There is a state migration from actor v3 to v4 in this upgrade, but we do not expect any network service outage. However, due to the short upgrade window, there may be some slowdown in block production as miners upgrade their software. To actively monitor the network situation, a 12-hour network maintenance window from 10 pm UTC Apr 28th to 10 am UTC Apr 29th is scheduled around the upgrade.