Do you trust Natively levels?

[DuoUnderscore]

What makes the grading system good is that it’s spread over a lot of users.

For many books this is the case, luckily. For many others it can be rather wild.

I do think you’re in the minority with the less digital lookups. On the Japanese side, it’s even a bit hard to find people who are reading extensively.

@HopeWaterfall

And I do think that the fact that a book is conceptually difficult should be taken into account for the grading. If the story is super straightforward, you have more room for misunderstanding things. Take a book where the narrator switches its point of view, from one character, one place and one time to another constantly, now if you misunderstand just one paragraph you can end up being completely lost. That’s also why children’s books are usually a lot easier than novels aimed at adults, it’s not just a matter of vocab, but also the concepts being more straightforward.

Well said. Yes this is important to consider when looking at overall difficulty, but I feel it kind of defeats the purpose of a website whose goal is to show you linguistically level appropriate material when external factors influence ratings more than the linguistic difficulty.

[seanblue]

@DuoUnderscore

I’d bet that the majority of users are probably reading digitally with a hover or tap dictionary. So, when they go to rate a book’s difficulty against another book’s, this hover dictionary user might not even consider that one book had way more vocab, more niche vocab, more spread vocab, and just rate them the same because the reader’s own level wasn’t even a factor in the difficulty.

Speaking personally, this is definitely not the case. I read on Kindle Paperwhite and use the built in dictionary, but I promise you there’s a perceptible difference when I have to look up a lot more words. It’s not perfect, and I may rate two books as the same when one requires some additional lookups, but if the number of lookups is noticeably greater it impacts my reading pace, my comprehension, and causes more/faster mental fatigue. All of these would cause me to rate the book requiring a lot more lookups as more difficult.

@DuoUnderscore

but I feel it kind of defeats the purpose of a website whose goal is to show you linguistically level appropriate material when external factors influence ratings more than the linguistic difficulty.

I think those things are still relevant to language learning though. Things that wouldn’t confuse you in your native language may confuse you in a second language if you aren’t high enough level. Your lack of understanding of the language can lead to confusion that makes things more difficult.

[HopeWaterfall]

Since I’m kinda curious now:

How do you usually do your lookups?

• Quick digital lookups
• Manual lookups
• It depends

How often do you usually do lookups?

• Almost every single new word
• A couple here and there
• Hardly ever
• It depends

[DuoUnderscore]

@seanblue

Speaking personally, this is definitely not the case. I read on Kindle Paperwhite and use the built in dictionary, but I promise you there’s a perceptible difference when I have to look up a lot more words

Yeah, there’s a difference and you can tell generally from the volume of lookups how many unknown words you had, but my point is that you do miss out on other aspects that make up difficulty. Bits like noticing, “oh this word isn’t normally used like this, I wonder if there’s a pun or something else the author’s trying to draw attention to”, or where there’s a character being silly and using uncommon but unimportant words, or where subtext is being implied. And those things really only come with reading material relatively close to one’s level. This gets back to my main point (which I should go ahead and be more direct about now), that tools are enabling people to consume content so far above their level that they’re often not actually qualified to rate them. Earlier on I was definitely a part of this problem when I was doing the same thing, so it’s not lost on me that it’s really easy to do, but it is important to realize that some of the ratings are unreliable and this is a large contributor to that.

[shitsurei]

Some context for my answers:

I don’t usually set out to do extensive or intensive reading, but the more comfortable I am with material the more likely I am to skip lookups and just keep going. I’ve also been burnt enough times by looking something up just to have it restated in the next sentence.

If I’m pushing my level I will usually look up every single word, even words that I have a petty good feeling for but am not 100% on.

I feel like I can easily grade those type of readings against each other.

@DuoUnderscore

The whole system is extremely susceptible to white noise, and while beginners-low intermediates should be encouraged to immerse and participate, they just throw off the accuracy of it all because they aren’t good enough judges of the difficulty of content that is significantly above their level.

I think this is quite the overstatement. While I quibble with the stack ranking, if I approach content within a given level range, it’s often a similar experience.

I also think that given a large enough group of graders that it doesn’t matter how they consume content, the ratings will be normalized into scores that correspond for how you consume content. So if the site tends to undergrade things in your view then you will just need to pick easier things. The fewer people grade things the more likely the samples are biased to a type of reader, but given enough votes, this should approach its “real” rating.

[mic]

I am not sure what to vote: I do digital look-ups but I have a physical text. So it’s not as fast as it could be with a pop-up dict, but also not as slow as it could be with a physical dict.

[seanblue]

@DuoUnderscore

tools are enabling people to consume content so far above their level that they’re often not actually qualified to rate them

That’s fair, to an extent. If your real reading level is 30, you probably can’t tell the difference between L35 and L37. But you probably can still tell the difference between L32 and L37, or L35 and L40. So while I think your general point makes sense, I wonder how much it actually hurts the gradings in practice (particularly for books that have 5+ users grading them).

That said, I don’t think I fall into this group because I have a low tolerance for pain and don’t read that far above my level very often. I think my current real reading level is around L30, and don’t really read above L35 (and don’t even go that high very often). Overreliance on Kindle lookups (and just generally being too busy/tired to do dedicated studying these days) does keep me from getting to a higher level though.

[pm215]

@mic

I am not sure what to vote: I do digital look-ups but I have a physical text

That’s “manual lookup” in my opinion. Nobody’s using a paper dictionary, I hope, so anything that’s not “click on the word and definition appears” is manual.

[DuoUnderscore]

Yes for what it’s worth I don’t mean to suggest that you personally are doing this and contributing bad ratings, I just happen to know newer people in various discord communities who are definitely doing this and contributing questionable ratings. On a large enough scale with users at levels with a high enough spread, it levels out, but there is a decent number of books that don’t get that treatment, and the issue becomes very apparent in the Video section.

[mic]

I also have a paper dict but I can only bear using it if look-ups turn out to be up to one item per page, which is still a rare situation.

[seanblue]

@HopeWaterfall

How often do you usually do lookups?

My answer is “it depends”, but for weird reasons. There are plenty of words I don’t know but don’t look up since they are obvious from context. But I also have a bad habit of second guessing myself, wanting to see if there’s specific nuance, or getting curious about the pitch accent, so I look up many words unnecessarily.

@DuoUnderscore

Yes for what it’s worth I don’t mean to suggest that you personally are doing this and contributing bad ratings

Yeah, I know. Just using myself as an example to say “not everyone does this” even if there are people who do.

@DuoUnderscore

On a large enough scale with users at levels with a high enough spread, it levels out, but there is a decent number of books that don’t get that treatment, and the issue becomes very apparent in the Video section.

Yeah, unfortunately there will always be niche books on a language learning site that get fewer ratings and are at greater risk of the problems you mention. For example, I read a lot of 4-koma manga and very often I’m the only one grading them, which isn’t super helpful.

I do find your commentary on videos particularly interesting as well. I probably have contributed to some of the problem there since I’ve watched several anime way above my level (with Japanese subs) and got by simply because I’d already watched them with English subs previously. For example, Vivy is L33, Saiki K is L32, Oshi no Ko is L30, and Violet Evergarden is L28, but I can’t say I really have a good sense of their relative difficulties from personal experience (and my real level for listening is probably closer to L25). (My gradings don’t seem wrong since it looks like I have apparently mostly graded consistently with the consensus, but I’m not sure I could describe to you why one is harder than another.)

[Naphthalene]

@HopeWaterfall

How do you usually do your lookups?

I do slow digital look ups by using my digital dictionary (it’s just the ebook version of the dictionary, but the search bar means it’s faster to look things up) whenever I can.
However, when I am reading with the booklive app, I am just using the built in dictionary because getting out of the app would be annoying.
… that being said, that point is mostly moot since I look up 0~2 words per book…

[Megumin]

That’s a both It depends.

For the first question, it will depend if I’m doing physical or digital reading, and the source.

For the second one, it will depend on the book.

[seanblue]

@Naphthalene

that being said, that point is mostly moot since I look up 0~2 words per book…

Funny, I think I looked up 5-10 words when reading The Way of Kings in English a few weeks back. Which is a great example of why easy lookups can be counterproductive, because I didn’t need to look up the words so much as I got curious even though I would never have bothered if I was reading physically.

[bibliothecary]

Gradings are subjective; there’s no escaping that. Even those who attend the same classes, use the same learning materials, and spend the same time studying will have differing opinions on how difficult something is. Natively uses the wisdom of the crowd to gauge difficulty, so the more people contribute, the more accurate the level. It’s not a rigorous academic study where you can control who reads what.

@DuoUnderscore

ratings are based on how hard it felt rather than how hard it was

Who’s to say how difficult something “really” is? Everyone (including you) is judging by their own perception of how difficult it felt to them. You can take different aspects of the language into consideration, but this doesn’t negate the fact that it’s subjective.

@DuoUnderscore

stories that are more visual/plot based are generally rated as easier

Visual storytelling helps provide context, which naturally aids understanding; should those who read novels disregard the plot because the natural patterns found in narratives give contextual clues that help with understanding the language?

@DuoUnderscore

it gets rated (artificially) as closer in difficulty than it should

This is your opinion - there is no objective measurement of difficulty that makes you right and others wrong.

@DuoUnderscore

while beginners-low intermediates should be encouraged to immerse and participate, they just throw off the accuracy of it

Are you proposing that only advanced learners be allowed to grade items? Then exceptionally few items suitable for beginners and intermediate would have gradings at all, not to mention you’d be excluding the people who could actually give feedback on whether or not something is suitable for the level that they are at.

You could make a whole list of things you think people should or shouldn’t take into account when grading, but it’s hardly as if you could enforce this. You could make a rule that only those who provide proof of having passed the JLPT N1 are allowed to grade items, but then there would be very few items graded, most of which would only be graded by one person. And how would you be able to tell if they’ve judged it well? How would you know they even read the book at all? I could go on, but you get the idea.

While I can agree on some of the issues you raise (I think Detailed language evaluations would be immensely useful for providing more information about what makes something difficult), I think a lot of what you’re unhappy with just stems from the subjectivity of grading, which is not something that’s going to change - everyone has different learning methods, experiences, and opinions, and this affects how they grade item difficulty.

Rather than being a flaw, I think it’s more beneficial to have a wide variety of experiences, especially if people include reviews - it makes the grading result more accurate and is helpful to more people. You could have an expert draft a list of media in order of difficulty according to their professional opinion, or have students taking the same class all read and grade the same items, or even create something similar to JPDB that takes more than just vocabulary into account… But none of these would be able to produce objective difficulty levels, either. And they certainly wouldn’t provide the volume of information that Natively does - over 12k Japanese books have non-temporary ratings.

One last thought: Natively is merely a helpful guide, it isn’t claiming to be a definitive source of the objective difficulty of foreign-language media. Difficulty levels fluctuate, and people can disagree with the level of even the most graded items, but that doesn’t detract from how useful it can be for finding things to enjoy at whatever level you are at in the language you are studying.

[DuoUnderscore]

@bibliothecary

Are you proposing that only advanced learners be allowed to grade items?

I’m not proposing anything. Realistically, there is no solution to any of these problems. The question is “Do you trust Natively levels” and this is just some food for thought about why it might not be quite as good as a lot of people here say. It isn’t a bad site or system, it just has some issues to be wary of.

@bibliothecary

Visual storytelling helps provide context, which naturally aids understanding; should those who read novels disregard the plot because the natural patterns found in narratives give contextual clues that help with understanding the language?

The point wasn’t that visuals don’t help or are bad to consider, it’s that when one’s comprehension is significantly below the material, they can convince themselves that something was easier than some other thing just because it had cool pictures to follow along with. That they would have come to the same conclusion had there been no words at all. I’m tired of hearing people weeks into studying on discord telling other beginners that so-and-so anime is great for beginners or that this-and-that other series is bad for beginners when the person giving this advice does not understand enough of either show to make that judgment reliably.

I’m all for subjectivity, but let’s be honest, some gradings lower the quality of the results.

[Psdes]

Great discussion here! I’ve always thought about the levels but was too lazy to make a thread about it.

As far as lookups go, I intentionally choose books where at most i’ll have to look up say 2 or 3 words per 100 words. I keep track of my known vocab which helps me choose the right books, but if there’s a book I want to read that requires more vocab, I usually just learn the extra vocab through SRS before I start reading.

We’re all different here but I personally really cannot tolerate looking up so many words while I read, it takes me out of the experience, I don’t want to feel like I’m learning a language while I read.

I havent graded anything in awhile, because I added all the anime I watched in one go, now I have like 500+ gradings and I just cant be bothered, nor can I remember the difficulty.

[bibliothecary]

@DuoUnderscore

I’m tired of hearing people weeks into studying on discord telling other beginners that so-and-so anime is great for beginners

Conversely, some advanced learners suggest items for beginners because it seems easy to them now, rather than considering how difficult they found it at the time.

But what harm is done if someone watches or reads something above their level, whether it was suggested to them by someone else or not? If they enjoy it, great - if not, they can just choose something else. Better to have a lot of suggestions than none at all, even if they’re not perfect.

@DuoUnderscore

I’m all for subjectivity, but let’s be honest, some gradings lower the quality of the results.

But it’s not as though Natively provides detailed rules for grading, it’s merely “what did you find more difficult?”, and all people can do is answer honestly based on their own knowledge and perceptions. As a learner, it’s natural there are things one doesn’t know (and doesn’t know they don’t know ) - I don’t see how that can affect the “quality” unless you’re only interested in the opinions of specific people (advanced learners, for example).

[mic]

@Psdes

2 or 3 words per 100 words

words → pages? There must be an error here, as this would mean 12-15 words per page. The rest does not sound like you could stand that, even though there are people that can.

[HopeWaterfall]

@bibliothecary

As a learner, it’s natural there are things one doesn’t know (and doesn’t know they don’t know ) - I don’t see how that can affect the “quality” unless you’re only interested in the opinions of specific people (advanced learners, for example).

I also feel like it’s great to have gradings from both beginners and more advanced learners. Everyone will notice different things, so that way we get an average level that encompasses as much as possible (from someone reading above their level that misses a lot of things but still manages to follow along, to someone for whom this is super easy to read). It also means that the levels will still make sense and be helpful if you read above/below your level.

Reading above your level, is, after all, an integral part of language learning. You have to do it at some point, and then that level that was once hard becomes easier. So it’s definitely valuable to have levels that take into account the experience of those that read above their level as well.